Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12723

S3A: Add ability to plug in any AWSCredentialsProvider

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Users can integrate a custom credential provider with S3A. See documentation of configuration property fs.s3a.aws.credentials.provider for further details.

      Description

      Although S3A currently has built-in support for org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider, com.amazonaws.auth.InstanceProfileCredentialsProvider, and org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider, it does not support any other credentials provider that implements the com.amazonaws.auth.AWSCredentialsProvider interface. Supporting the ability to plug in any com.amazonaws.auth.AWSCredentialsProvider instance will expand the options for S3 credentials, such as:

      • temporary credentials from STS, e.g. via com.amazonaws.auth.STSSessionCredentialsProvider
      • IAM role-based credentials, e.g. via com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider
      • a custom credentials provider that satisfies one's own needs, e.g. bucket-specific credentials, user-specific credentials, etc.

      To support this, we can add a configuration for the fully qualified class name of a credentials provider, to be loaded by S3AFileSystem.initialize(URI, Configuration).

      The configured credentials provider should implement com.amazonaws.auth.AWSCredentialsProvider and have a constructor that accepts (URI uri, Configuration conf).

      1. HADOOP-12723.0.patch
        9 kB
        Steven K. Wong
      2. HADOOP-12723.1.patch
        9 kB
        Steven K. Wong
      3. HADOOP-12723.2.patch
        9 kB
        Steven K. Wong
      4. HADOOP-12723.3.patch
        10 kB
        Steven K. Wong
      5. HADOOP-12723.4.patch
        10 kB
        Steven K. Wong
      6. HADOOP-12723.5.patch
        13 kB
        Steven K. Wong
      7. HADOOP-12723-branch-2-006.patch
        13 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Related to HADOOP-12537, though wants to be more generic. Making it a full plugin point would be more flexible in future

          Show
          stevel@apache.org Steve Loughran added a comment - Related to HADOOP-12537 , though wants to be more generic. Making it a full plugin point would be more flexible in future
          Hide
          slider Steven K. Wong added a comment -

          I have looked at HADOOP-12537. It is good when you want to supply the key/secret/token in the configuration and need S3 access only for a short period of time, because the validity of the key/secret/token is short-lived. In contrast, plugging in a credentials provider like STSSessionCredentialsProvider allows continued S3 access for as long as needed because it auto-refreshes the key/secret/token.

          Show
          slider Steven K. Wong added a comment - I have looked at HADOOP-12537 . It is good when you want to supply the key/secret/token in the configuration and need S3 access only for a short period of time, because the validity of the key/secret/token is short-lived. In contrast, plugging in a credentials provider like STSSessionCredentialsProvider allows continued S3 access for as long as needed because it auto-refreshes the key/secret/token.
          Hide
          slider Steven K. Wong added a comment -

          Attaching patch. I tested it in the us-east-1 region.

          Show
          slider Steven K. Wong added a comment - Attaching patch. I tested it in the us-east-1 region.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 3s Maven dependency ordering for branch
          +1 mvninstall 6m 53s trunk passed
          +1 compile 6m 0s trunk passed with JDK v1.8.0_66
          +1 compile 6m 49s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 1m 21s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 18s trunk passed
          +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 5m 52s the patch passed with JDK v1.8.0_66
          +1 javac 5m 52s the patch passed
          +1 compile 6m 44s the patch passed with JDK v1.7.0_91
          +1 javac 6m 44s the patch passed
          +1 checkstyle 1m 4s root: patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 47s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91
          -1 unit 6m 58s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 5s hadoop-common in the patch passed with JDK v1.7.0_91.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          66m 49s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.ipc.TestRPCWaitForProxy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784975/HADOOP-12723.0.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux bdb6544d9876 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef343be
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Max memory used 77MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 3s Maven dependency ordering for branch +1 mvninstall 6m 53s trunk passed +1 compile 6m 0s trunk passed with JDK v1.8.0_66 +1 compile 6m 49s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 18s trunk passed +1 javadoc 1m 5s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 17s trunk passed with JDK v1.7.0_91 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 5m 52s the patch passed with JDK v1.8.0_66 +1 javac 5m 52s the patch passed +1 compile 6m 44s the patch passed with JDK v1.7.0_91 +1 javac 6m 44s the patch passed +1 checkstyle 1m 4s root: patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 47s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 17s the patch passed with JDK v1.7.0_91 -1 unit 6m 58s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_66. +1 unit 7m 5s hadoop-common in the patch passed with JDK v1.7.0_91. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 66m 49s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.ipc.TestRPCWaitForProxy Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12784975/HADOOP-12723.0.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux bdb6544d9876 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef343be Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Max memory used 77MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8487/console This message was automatically generated.
          Hide
          slider Steven K. Wong added a comment -

          The failed junit tests appear to be unrelated.

          Show
          slider Steven K. Wong added a comment - The failed junit tests appear to be unrelated.
          Hide
          slider Steven K. Wong added a comment -

          Is there anything else needed before a committer will review the patch? Please let me know if there is. I'm assuming folks are just busy. Thanks.

          Show
          slider Steven K. Wong added a comment - Is there anything else needed before a committer will review the patch? Please let me know if there is. I'm assuming folks are just busy. Thanks.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Busy; the usuaI. If it makes you feel better I suffer from a lack of review of my own patches (YARN-679 anyone?) —if its not considered critical not enough people put the time in. I do feel you pain

          I was actually running through the fs/s3 stuff to see if there was anything easy to pull in; I missed this one as it wasn't linked too off HADOOP-11694; I'll fix that now.

          I looked at HADOOP-12537, which is the related one, and saw that it added a new JAR to list. as soon as I saw that I felt that it wasn't something for a last-minute patch, it'd need more experimentation (specifically, what would happen if the new JAR —listed as for test only— wasn't there?)

          Show
          stevel@apache.org Steve Loughran added a comment - Busy; the usuaI. If it makes you feel better I suffer from a lack of review of my own patches ( YARN-679 anyone?) —if its not considered critical not enough people put the time in. I do feel you pain I was actually running through the fs/s3 stuff to see if there was anything easy to pull in; I missed this one as it wasn't linked too off HADOOP-11694 ; I'll fix that now. I looked at HADOOP-12537 , which is the related one, and saw that it added a new JAR to list. as soon as I saw that I felt that it wasn't something for a last-minute patch, it'd need more experimentation (specifically, what would happen if the new JAR —listed as for test only— wasn't there?)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Can anyone other than me play with this patch to see how well it works out?

          Show
          stevel@apache.org Steve Loughran added a comment - Can anyone other than me play with this patch to see how well it works out?
          Hide
          slider Steven K. Wong added a comment -

          Rebasing the patch.

          Show
          slider Steven K. Wong added a comment - Rebasing the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 6m 33s trunk passed
          +1 compile 6m 1s trunk passed with JDK v1.8.0_77
          +1 compile 6m 44s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 4s trunk passed
          +1 mvnsite 1m 15s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 2m 29s trunk passed
          +1 javadoc 1m 19s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 29s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 7m 9s the patch passed with JDK v1.8.0_77
          +1 javac 7m 9s the patch passed
          +1 compile 7m 27s the patch passed with JDK v1.7.0_95
          +1 javac 7m 27s the patch passed
          +1 checkstyle 1m 8s the patch passed
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 29s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 39s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 25s the patch passed with JDK v1.7.0_95
          +1 unit 8m 53s hadoop-common in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_77.
          +1 unit 8m 46s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          71m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797953/HADOOP-12723.1.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux ce5d57b5a42b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1b78b2b
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9056/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9056/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 6m 33s trunk passed +1 compile 6m 1s trunk passed with JDK v1.8.0_77 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 4s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 19s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 29s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 7m 9s the patch passed with JDK v1.8.0_77 +1 javac 7m 9s the patch passed +1 compile 7m 27s the patch passed with JDK v1.7.0_95 +1 javac 7m 27s the patch passed +1 checkstyle 1m 8s the patch passed +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 39s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 25s the patch passed with JDK v1.7.0_95 +1 unit 8m 53s hadoop-common in the patch passed with JDK v1.8.0_77. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_77. +1 unit 8m 46s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 71m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797953/HADOOP-12723.1.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux ce5d57b5a42b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1b78b2b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9056/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9056/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. I'd prefer using a commons.lang check for string null/emptyness, rather than com.google.common.base.Strings ... our relationship with guava is, well, strained.
          2. Id also like those caught and rethrown exceptions of construction wrapped with an IOE, rather than RTE. It's a convention in the hadoop FS layer that most methods declare they throw IOE, clients except to catch them for problems.
          Show
          stevel@apache.org Steve Loughran added a comment - I'd prefer using a commons.lang check for string null/emptyness, rather than com.google.common.base.Strings ... our relationship with guava is, well, strained. Id also like those caught and rethrown exceptions of construction wrapped with an IOE, rather than RTE. It's a convention in the hadoop FS layer that most methods declare they throw IOE, clients except to catch them for problems.
          Hide
          slider Steven K. Wong added a comment -

          Steve Loughran, thanks for reviewing this. Attaching a new patch to address your comments.

          Show
          slider Steven K. Wong added a comment - Steve Loughran , thanks for reviewing this. Attaching a new patch to address your comments.
          Hide
          slider Steven K. Wong added a comment -

          Rebasing the patch. Also, fix two toString calls inside S3AFileSystem.toString, one of which causes NPE when there's no canned ACL.

          Show
          slider Steven K. Wong added a comment - Rebasing the patch. Also, fix two toString calls inside S3AFileSystem.toString, one of which causes NPE when there's no canned ACL.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 6m 27s trunk passed with JDK v1.8.0_91
          +1 compile 7m 5s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 1m 23s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 7s trunk passed with JDK v1.8.0_91
          +1 javadoc 1m 20s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 6m 10s the patch passed with JDK v1.8.0_91
          +1 javac 6m 10s the patch passed
          +1 compile 7m 0s the patch passed with JDK v1.7.0_95
          +1 javac 7m 0s the patch passed
          -1 checkstyle 1m 27s root: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 43s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 24s the patch passed with JDK v1.7.0_95
          +1 unit 8m 37s hadoop-common in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          70m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:cf2ee45
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803939/HADOOP-12723.3.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 09075a4aa546 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3fa1380
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 6m 27s trunk passed with JDK v1.8.0_91 +1 compile 7m 5s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 23s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 7s trunk passed with JDK v1.8.0_91 +1 javadoc 1m 20s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 6m 10s the patch passed with JDK v1.8.0_91 +1 javac 6m 10s the patch passed +1 compile 7m 0s the patch passed with JDK v1.7.0_95 +1 javac 7m 0s the patch passed -1 checkstyle 1m 27s root: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 43s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 24s the patch passed with JDK v1.7.0_95 +1 unit 8m 37s hadoop-common in the patch passed with JDK v1.8.0_91. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 70m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12803939/HADOOP-12723.3.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 09075a4aa546 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fa1380 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9419/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          slider Steven K. Wong added a comment -

          The new Checkstyle warning "Missing package-info.java file" seems innocuous.

          Show
          slider Steven K. Wong added a comment - The new Checkstyle warning "Missing package-info.java file" seems innocuous.
          Hide
          slider Steven K. Wong added a comment -

          Rebasing the patch. Could someone review it again?

          Show
          slider Steven K. Wong added a comment - Rebasing the patch. Could someone review it again?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 11s trunk passed
          +1 compile 6m 30s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 9s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 1m 46s trunk passed
          +1 javadoc 1m 8s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 0m 50s the patch passed
          +1 compile 6m 31s the patch passed
          +1 javac 6m 31s the patch passed
          +1 checkstyle 1m 28s the patch passed
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 55s the patch passed
          +1 javadoc 1m 17s the patch passed
          +1 unit 8m 24s hadoop-common in the patch passed.
          +1 unit 0m 14s hadoop-aws in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          42m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804541/HADOOP-12723.4.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9b98ece2b3e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dd99f5f
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9475/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9475/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 11s trunk passed +1 compile 6m 30s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 9s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 1m 46s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 6m 31s the patch passed +1 javac 6m 31s the patch passed +1 checkstyle 1m 28s the patch passed +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 55s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 8m 24s hadoop-common in the patch passed. +1 unit 0m 14s hadoop-aws in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 42m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804541/HADOOP-12723.4.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9b98ece2b3e3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dd99f5f Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9475/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9475/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. The patch reverts a couple of things in the toString() call. I'll go with line 1395, but not sure about the L1383 change; the existing code omits the line completely.
          2. getAWSCredentialsProvider : need to have consensus there on what classloader to use
          3. Once HADOOP-13130 gets in then it'll break TestS3AAWSCredentialsProvider; all Amazon exceptions will be caught and wrapped by IOEs; your test will need to change. It'll be
            a race to see whose gets in first...
          4. TestS3AAWSCredentialsProvider: codepaths that expected to raise an exception should have a fail(message) on the path which is expected to fail. Otherwise test failures could be missed
          5. you'll need to cover the new properties in the AWS site/index.md file.

          One question I have here is how would this relate to HADOOP-12537; STS support? I believe That mandates the STS JAR on the CP, and adds extra config options for the binding. If it's the latter, the binding info, this means that your patch isn't going to be sufficient on its own. What would it take to support it?

          Presumably, it'd need a new provider which looked up the session key from the properties, yes? If so, that'd be the first one to add after this patch, marking the STS JAR as a "provided" dependency of hadoop-aws.

          What other providers do you have in mind?

          Show
          stevel@apache.org Steve Loughran added a comment - The patch reverts a couple of things in the toString() call. I'll go with line 1395, but not sure about the L1383 change; the existing code omits the line completely. getAWSCredentialsProvider : need to have consensus there on what classloader to use Once HADOOP-13130 gets in then it'll break TestS3AAWSCredentialsProvider ; all Amazon exceptions will be caught and wrapped by IOEs; your test will need to change. It'll be a race to see whose gets in first... TestS3AAWSCredentialsProvider : codepaths that expected to raise an exception should have a fail(message) on the path which is expected to fail. Otherwise test failures could be missed you'll need to cover the new properties in the AWS site/index.md file. One question I have here is how would this relate to HADOOP-12537 ; STS support? I believe That mandates the STS JAR on the CP, and adds extra config options for the binding. If it's the latter, the binding info, this means that your patch isn't going to be sufficient on its own. What would it take to support it? Presumably, it'd need a new provider which looked up the session key from the properties, yes? If so, that'd be the first one to add after this patch, marking the STS JAR as a "provided" dependency of hadoop-aws. What other providers do you have in mind?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          (we all found that NPE BTW)

          Show
          stevel@apache.org Steve Loughran added a comment - (we all found that NPE BTW)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          PS, I'm being strict about S3 Patches: you do have to declare that you've run all the aws tests against an S3 Implementation, and state which AWS location you worked with; if some in-house thing, declare that + also run against AWS. This is our due diligence filter; it takes long enough for those test to complete that it kills the review process for the reviewer to do all the work. sorry

          https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure

          Show
          stevel@apache.org Steve Loughran added a comment - PS, I'm being strict about S3 Patches: you do have to declare that you've run all the aws tests against an S3 Implementation, and state which AWS location you worked with; if some in-house thing, declare that + also run against AWS. This is our due diligence filter; it takes long enough for those test to complete that it kills the review process for the reviewer to do all the work. sorry https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure
          Hide
          slider Steven K. Wong added a comment -

          Steve Loughran, thanks for taking time to review the patch.

          Attaching HADOOP-12723.5.patch that accepts all your suggestions, except for #2 which I'm unclear about what you meant.

          Regarding HADOOP-12537, an earlier comment of mine (above) compares it with an alternative with different functionality (short-lived vs. long-lived credentials). But, yes, one can implement HADOOP-12537 using HADOOP-12723: Implement and plug in a new provider that, just like HADOOP-12537 currently does, looks up the session token from configuration and returns a com.amazonaws.auth.BasicSessionCredentials.

          Regarding what other providers I have in mind, I have seen a use case for a provider that doles out credentials using com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider with a role ARN (specified in a config property), and a use case for doling out different role-based credentials based on the S3 bucket being accessed (using config properties that map bucket names to role ARNs).

          Show
          slider Steven K. Wong added a comment - Steve Loughran , thanks for taking time to review the patch. Attaching HADOOP-12723 .5.patch that accepts all your suggestions, except for #2 which I'm unclear about what you meant. Regarding HADOOP-12537 , an earlier comment of mine (above) compares it with an alternative with different functionality (short-lived vs. long-lived credentials). But, yes, one can implement HADOOP-12537 using HADOOP-12723 : Implement and plug in a new provider that, just like HADOOP-12537 currently does, looks up the session token from configuration and returns a com.amazonaws.auth.BasicSessionCredentials. Regarding what other providers I have in mind, I have seen a use case for a provider that doles out credentials using com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider with a role ARN (specified in a config property), and a use case for doling out different role-based credentials based on the S3 bucket being accessed (using config properties that map bucket names to role ARNs).
          Hide
          slider Steven K. Wong added a comment -

          HADOOP-12723.5.patch is tested against S3 Standard from us-east-1, including running all TestS3A* unit tests.

          Show
          slider Steven K. Wong added a comment - HADOOP-12723 .5.patch is tested against S3 Standard from us-east-1, including running all TestS3A* unit tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 56s Maven dependency ordering for branch
          +1 mvninstall 6m 13s trunk passed
          +1 compile 6m 27s trunk passed
          +1 checkstyle 1m 23s trunk passed
          +1 mvnsite 1m 12s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 0m 50s the patch passed
          +1 compile 7m 31s the patch passed
          +1 javac 7m 31s the patch passed
          -1 checkstyle 1m 23s root: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 21s the patch passed
          +1 javadoc 1m 12s the patch passed
          +1 unit 8m 52s hadoop-common in the patch passed.
          +1 unit 0m 14s hadoop-aws in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          45m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804818/HADOOP-12723.5.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 9f5e00809753 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 010e6ac
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 56s Maven dependency ordering for branch +1 mvninstall 6m 13s trunk passed +1 compile 6m 27s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 0m 50s the patch passed +1 compile 7m 31s the patch passed +1 javac 7m 31s the patch passed -1 checkstyle 1m 23s root: The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 21s the patch passed +1 javadoc 1m 12s the patch passed +1 unit 8m 52s hadoop-common in the patch passed. +1 unit 0m 14s hadoop-aws in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 45m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804818/HADOOP-12723.5.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9f5e00809753 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 010e6ac Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9496/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          slider Steven K. Wong added a comment -

          It's the "Missing package-info.java file" again.

          Show
          slider Steven K. Wong added a comment - It's the "Missing package-info.java file" again.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 28s trunk passed
          +1 compile 6m 29s trunk passed
          +1 checkstyle 1m 24s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 1m 1s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 1m 14s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 7m 24s the patch passed
          +1 javac 7m 24s the patch passed
          +1 checkstyle 1m 28s the patch passed
          +1 mvnsite 1m 15s the patch passed
          +1 mvneclipse 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 8s the patch passed
          +1 unit 7m 52s hadoop-common in the patch passed.
          +1 unit 0m 14s hadoop-aws in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          45m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804818/HADOOP-12723.5.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 7887071a89bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1597630
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9501/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9501/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 38s Maven dependency ordering for branch +1 mvninstall 6m 28s trunk passed +1 compile 6m 29s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 1m 1s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 1m 14s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 7m 24s the patch passed +1 javac 7m 24s the patch passed +1 checkstyle 1m 28s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 8s the patch passed +1 unit 7m 52s hadoop-common in the patch passed. +1 unit 0m 14s hadoop-aws in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 45m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804818/HADOOP-12723.5.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 7887071a89bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1597630 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9501/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9501/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is patch 006; what I'll merge in once my local test run is happy.

          Changes from 005

          1. javadoc on new method
          2. use Configuration.getTrimmed() to get the config. This should be the default for all string getters these days.
          3. tests will rethrow the exception if it is the wrong kind —avoids failure details being lost.
          Show
          stevel@apache.org Steve Loughran added a comment - This is patch 006; what I'll merge in once my local test run is happy. Changes from 005 javadoc on new method use Configuration.getTrimmed() to get the config. This should be the default for all string getters these days. tests will rethrow the exception if it is the wrong kind —avoids failure details being lost.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 57s Maven dependency ordering for branch
          +1 mvninstall 6m 31s branch-2 passed
          +1 compile 5m 38s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 13s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 21s branch-2 passed
          +1 mvnsite 1m 13s branch-2 passed
          +1 mvneclipse 0m 29s branch-2 passed
          +1 findbugs 2m 3s branch-2 passed
          +1 javadoc 1m 7s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 19s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 0m 53s the patch passed
          +1 compile 5m 32s the patch passed with JDK v1.8.0_91
          +1 javac 5m 32s the patch passed
          +1 compile 6m 17s the patch passed with JDK v1.7.0_101
          +1 javac 6m 17s the patch passed
          -1 checkstyle 1m 28s root: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21)
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 55s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 29s the patch passed with JDK v1.7.0_101
          -1 unit 9m 5s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_91.
          -1 unit 8m 59s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          -1 asflicense 0m 23s The patch generated 1 ASF License warnings.
          69m 25s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.7.0_101 Failed junit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805188/HADOOP-12723-branch-2-006.patch
          JIRA Issue HADOOP-12723
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 49afdb239197 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e7f396f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 57s Maven dependency ordering for branch +1 mvninstall 6m 31s branch-2 passed +1 compile 5m 38s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 13s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 21s branch-2 passed +1 mvnsite 1m 13s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed +1 findbugs 2m 3s branch-2 passed +1 javadoc 1m 7s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 19s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 0m 53s the patch passed +1 compile 5m 32s the patch passed with JDK v1.8.0_91 +1 javac 5m 32s the patch passed +1 compile 6m 17s the patch passed with JDK v1.7.0_101 +1 javac 6m 17s the patch passed -1 checkstyle 1m 28s root: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 55s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 29s the patch passed with JDK v1.7.0_101 -1 unit 9m 5s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.8.0_91. -1 unit 8m 59s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. -1 asflicense 0m 23s The patch generated 1 ASF License warnings. 69m 25s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_101 Failed junit tests hadoop.security.token.delegation.TestZKDelegationTokenSecretManager Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805188/HADOOP-12723-branch-2-006.patch JIRA Issue HADOOP-12723 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 49afdb239197 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e7f396f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/testReport/ asflicense https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9534/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1

          committed, thanks for this.

          Show
          stevel@apache.org Steve Loughran added a comment - +1 committed, thanks for this.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9833 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9833/)
          HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. (stevel: rev 757050ff355d40bc28f9dbfd0c0083c5f337d270)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9833 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9833/ ) HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. (stevel: rev 757050ff355d40bc28f9dbfd0c0083c5f337d270) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note that it turns out that the bad config test was broken; after HADOOP-13058 FS.initalize() swallows access exceptions trying to set multipart thresholds, which was where the 403 was being triggered before. HADOOP-13130 patch 014 addresses that.

          Its always important to have the non-exception-throwing codepath to assert that it is never reached —that is what was happening here, hence why the tests weren't catching the regression in the test. Apologies for not noticing that during review

          Show
          stevel@apache.org Steve Loughran added a comment - Note that it turns out that the bad config test was broken; after HADOOP-13058 FS.initalize() swallows access exceptions trying to set multipart thresholds, which was where the 403 was being triggered before. HADOOP-13130 patch 014 addresses that. Its always important to have the non-exception-throwing codepath to assert that it is never reached —that is what was happening here, hence why the tests weren't catching the regression in the test. Apologies for not noticing that during review
          Hide
          slider Steven K. Wong added a comment -

          Thanks for reviewing and for pointing out the no-exception code path (my bad).

          Show
          slider Steven K. Wong added a comment - Thanks for reviewing and for pointing out the no-exception code path (my bad).

            People

            • Assignee:
              slider Steven K. Wong
              Reporter:
              slider Steven K. Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development