Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12537

S3A to support Amazon STS temporary credentials

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Release Note:
      S3A now supports use of AWS Security Token Service temporary credentials for authentication to S3. Refer to the documentation of configuration property fs.s3a.session.token for further details.

      Description

      Amazon STS allows you to issue temporary access key id / secret key pairs for your a user / role. However, using these credentials also requires specifying a session ID. There is currently no such configuration property or the required code to pass it through to the API (at least not that I can find) in any of the S3 connectors.

      1. HADOOP-12537.001.patch
        13 kB
        Sean Mackrory
      2. HADOOP-12537.002.patch
        14 kB
        Sean Mackrory
      3. HADOOP-12537.003.patch
        6 kB
        Sean Mackrory
      4. HADOOP-12537.004.patch
        14 kB
        Sean Mackrory
      5. HADOOP-12537.006.patch
        19 kB
        Sean Mackrory
      6. HADOOP-12537.diff
        10 kB
        Sean Mackrory
      7. HADOOP-12537.diff
        10 kB
        Sean Mackrory
      8. HADOOP-12537-branch-2.005.patch
        19 kB
        Steve Loughran
      9. HADOOP-12537-branch-2.007.patch
        19 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Sean, this must be for s3a only, clarifying in title

          Show
          stevel@apache.org Steve Loughran added a comment - Sean, this must be for s3a only, clarifying in title
          Hide
          mackrorysd Sean Mackrory added a comment -

          Thanks! s3a is the one I'm personally interested in most, but fwiw such a flag is not implemented for s3 and s3n. Are those connectors considered deprecated?

          Show
          mackrorysd Sean Mackrory added a comment - Thanks! s3a is the one I'm personally interested in most, but fwiw such a flag is not implemented for s3 and s3n. Are those connectors considered deprecated?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          s3: has been essentially deprecated for a long time, nobody uses it for new data

          s3n is considered stable, and after the disaster that was HADOOP-9623, in which a patch which swallowed exceptions snuck in, there's effectively a veto on any patch there that isn't considered critical in terms of security or performance. No new features.

          The failures that patch triggered (HADOOP-10589) showed that there wasn 't adequate testing of the s3 clients; that's better now, but it means there'll be an expectation of more tests for any feature; tests that are (a) robust even over long-haul connections (b) skip nicely against non-AWS implementations of the S3 APIs and (c) tested by you against multiple s3 endpoints, including the more recent ones with tighter auth (e.g. AWS frankfurt).

          Show
          stevel@apache.org Steve Loughran added a comment - s3: has been essentially deprecated for a long time, nobody uses it for new data s3n is considered stable, and after the disaster that was HADOOP-9623 , in which a patch which swallowed exceptions snuck in, there's effectively a veto on any patch there that isn't considered critical in terms of security or performance. No new features. The failures that patch triggered ( HADOOP-10589 ) showed that there wasn 't adequate testing of the s3 clients; that's better now, but it means there'll be an expectation of more tests for any feature; tests that are (a) robust even over long-haul connections (b) skip nicely against non-AWS implementations of the S3 APIs and (c) tested by you against multiple s3 endpoints, including the more recent ones with tighter auth (e.g. AWS frankfurt).
          Hide
          mackrorysd Sean Mackrory added a comment -

          Attaching a patch that I think may do the trick. I've tested it against several different S3 and STS endpoints (including Frankfurt in both cases). Not sure how best to make sure it's skipped when using a non-AWS option, although that is important since an STS implementation is probably uncommon outside of AWS too. Need to do a bit more testing tomorrow before I really think this is worth consideration, just posting what I have now in case I can't actually get to it tomorrow...

          Show
          mackrorysd Sean Mackrory added a comment - Attaching a patch that I think may do the trick. I've tested it against several different S3 and STS endpoints (including Frankfurt in both cases). Not sure how best to make sure it's skipped when using a non-AWS option, although that is important since an STS implementation is probably uncommon outside of AWS too. Need to do a bit more testing tomorrow before I really think this is worth consideration, just posting what I have now in case I can't actually get to it tomorrow...
          Hide
          mackrorysd Sean Mackrory added a comment -

          And thanks for the context, btw.

          Show
          mackrorysd Sean Mackrory added a comment - And thanks for the context, btw.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Restoring a couple of tests my patch was nuking for no good reason and adding Javadoc for the test.

          One other shortcoming I still need to address: one shouldn't have to provide AWS keys in auth-keys.xml to run these tests - it should be possible to run the tests using an EC2 instance profile instead. The test should allow for connecting to STS without explicit credentials if none are provided.

          Show
          mackrorysd Sean Mackrory added a comment - Restoring a couple of tests my patch was nuking for no good reason and adding Javadoc for the test. One other shortcoming I still need to address: one shouldn't have to provide AWS keys in auth-keys.xml to run these tests - it should be possible to run the tests using an EC2 instance profile instead. The test should allow for connecting to STS without explicit credentials if none are provided.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. why not make this a new test case class, one that skips (using Assume) if the STS config option is unset? The keys will still need to be in there to trigger the hadoop-aws test run. We could easily make the java4 tests skippable without auth, but the inherited FilesystemContractBaseTest code lacks that
          2. if you change the scope of the new STS lib to <provided>, it'll get pulled in for both build & test but not picked up.
          3. make sure your test dirs are cleaned up even if a test fails -put them in finally clauses or in an @After method.
          Show
          stevel@apache.org Steve Loughran added a comment - why not make this a new test case class, one that skips (using Assume ) if the STS config option is unset? The keys will still need to be in there to trigger the hadoop-aws test run. We could easily make the java4 tests skippable without auth, but the inherited FilesystemContractBaseTest code lacks that if you change the scope of the new STS lib to <provided>, it'll get pulled in for both build & test but not picked up. make sure your test dirs are cleaned up even if a test fails -put them in finally clauses or in an @After method.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Attaching a patch with tests that I've run against all S3 and STS endpoints, including combinations in different regions. I also tested this from the CLI and verified other authentication methods still work. TestAutomaticProxyPortSelection fails, but it was failing prior to my patch and all other tests in hadoop-aws pass. I moved the test into its own class, am doing clean up with @After, and skip the test if test.sts.enabled isn't set.

          The STS jar is not required during the build - only when running tests. The feature assumes the user is going to get the temporary credentials somewhere else. The test has to them itself.

          Show
          mackrorysd Sean Mackrory added a comment - Attaching a patch with tests that I've run against all S3 and STS endpoints, including combinations in different regions. I also tested this from the CLI and verified other authentication methods still work. TestAutomaticProxyPortSelection fails, but it was failing prior to my patch and all other tests in hadoop-aws pass. I moved the test into its own class, am doing clean up with @After, and skip the test if test.sts.enabled isn't set. The STS jar is not required during the build - only when running tests. The feature assumes the user is going to get the temporary credentials somewhere else. The test has to them itself.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          0 patch 0m 8s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 8s trunk passed
          +1 compile 4m 20s trunk passed with JDK v1.8.0_60
          +1 compile 4m 7s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 56s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 1m 11s trunk passed
          -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 26s trunk passed with JDK v1.7.0_79
          +1 mvninstall 2m 1s the patch passed
          +1 compile 4m 16s the patch passed with JDK v1.8.0_60
          +1 javac 4m 16s the patch passed
          +1 compile 4m 6s the patch passed with JDK v1.7.0_79
          +1 javac 4m 6s the patch passed
          -1 checkstyle 0m 58s Patch generated 2 new checkstyle issues in root (total was 62, now 63).
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 0m 11s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 28s the patch passed with JDK v1.7.0_79
          -1 unit 20m 38s hadoop-common in the patch failed with JDK v1.8.0_60.
          +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_60.
          -1 unit 6m 24s hadoop-common in the patch failed with JDK v1.7.0_79.
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.7.0_79.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 21s Patch does not generate ASF License warnings.
          67m 59s



          Reason Tests
          JDK v1.8.0_60 Failed junit tests hadoop.metrics2.sink.TestFileSink
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
          JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC
            hadoop.metrics2.sink.TestFileSink



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770431/HADOOP-12537.diff
          JIRA Issue HADOOP-12537
          Optional Tests asflicense javac javadoc mvninstall unit xml compile findbugs checkstyle site mvnsite
          uname Linux 2e38106bbc39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh
          git revision trunk / dac0463
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: .
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. 0 patch 0m 8s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 8s trunk passed +1 compile 4m 20s trunk passed with JDK v1.8.0_60 +1 compile 4m 7s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 1m 11s trunk passed -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 1m 12s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 26s trunk passed with JDK v1.7.0_79 +1 mvninstall 2m 1s the patch passed +1 compile 4m 16s the patch passed with JDK v1.8.0_60 +1 javac 4m 16s the patch passed +1 compile 4m 6s the patch passed with JDK v1.7.0_79 +1 javac 4m 6s the patch passed -1 checkstyle 0m 58s Patch generated 2 new checkstyle issues in root (total was 62, now 63). +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 0m 11s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 1m 12s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 28s the patch passed with JDK v1.7.0_79 -1 unit 20m 38s hadoop-common in the patch failed with JDK v1.8.0_60. +1 unit 0m 7s hadoop-project in the patch passed with JDK v1.8.0_60. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_60. -1 unit 6m 24s hadoop-common in the patch failed with JDK v1.7.0_79. +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.7.0_79. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 67m 59s Reason Tests JDK v1.8.0_60 Failed junit tests hadoop.metrics2.sink.TestFileSink JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestIPC   hadoop.metrics2.sink.TestFileSink Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770431/HADOOP-12537.diff JIRA Issue HADOOP-12537 Optional Tests asflicense javac javadoc mvninstall unit xml compile findbugs checkstyle site mvnsite uname Linux 2e38106bbc39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-1a9afee/precommit/personality/hadoop.sh git revision trunk / dac0463 Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: . Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8018/console This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Addressing the checkstyles and patch naming concerns. I do not believe the other concerns are related to my patch, but perhaps someone could enlighten me if I'm mistaken.

          Show
          mackrorysd Sean Mackrory added a comment - Addressing the checkstyles and patch naming concerns. I do not believe the other concerns are related to my patch, but perhaps someone could enlighten me if I'm mistaken.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 12s trunk passed
          +1 compile 4m 49s trunk passed with JDK v1.8.0_60
          +1 compile 4m 28s trunk passed with JDK v1.7.0_79
          +1 checkstyle 1m 0s trunk passed
          +1 mvnsite 1m 30s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 1m 20s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 33s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 56s the patch passed
          +1 compile 4m 37s the patch passed with JDK v1.8.0_60
          +1 javac 4m 37s the patch passed
          +1 compile 4m 32s the patch passed with JDK v1.7.0_79
          +1 javac 4m 32s the patch passed
          -1 checkstyle 1m 1s Patch generated 2 new checkstyle issues in root (total was 63, now 64).
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 0m 10s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 1m 19s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 33s the patch passed with JDK v1.7.0_79
          +1 unit 7m 12s hadoop-common in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_60.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60.
          +1 unit 7m 8s hadoop-common in the patch passed with JDK v1.7.0_79.
          +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.7.0_79.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          57m 49s



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770506/HADOOP-12537.001.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense javac javadoc mvninstall unit xml compile findbugs checkstyle site mvnsite
          uname Linux c63e2a4c7b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh
          git revision trunk / 194251c
          Default Java 1.7.0_79
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/artifact/patchprocess/diff-checkstyle-root.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: .
          Max memory used 225MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 12s trunk passed +1 compile 4m 49s trunk passed with JDK v1.8.0_60 +1 compile 4m 28s trunk passed with JDK v1.7.0_79 +1 checkstyle 1m 0s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 42s trunk passed -1 findbugs 0m 11s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 1m 20s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 33s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 56s the patch passed +1 compile 4m 37s the patch passed with JDK v1.8.0_60 +1 javac 4m 37s the patch passed +1 compile 4m 32s the patch passed with JDK v1.7.0_79 +1 javac 4m 32s the patch passed -1 checkstyle 1m 1s Patch generated 2 new checkstyle issues in root (total was 63, now 64). +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 0m 10s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 1m 19s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_79 +1 unit 7m 12s hadoop-common in the patch passed with JDK v1.8.0_60. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.8.0_60. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_60. +1 unit 7m 8s hadoop-common in the patch passed with JDK v1.7.0_79. +1 unit 0m 10s hadoop-project in the patch passed with JDK v1.7.0_79. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 57m 49s Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-04 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12770506/HADOOP-12537.001.patch JIRA Issue HADOOP-12537 Optional Tests asflicense javac javadoc mvninstall unit xml compile findbugs checkstyle site mvnsite uname Linux c63e2a4c7b1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-d0f6847/precommit/personality/hadoop.sh git revision trunk / 194251c Default Java 1.7.0_79 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_60 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_79 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/artifact/patchprocess/diff-checkstyle-root.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: . Max memory used 225MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8025/console This message was automatically generated.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Hi, Sean Mackrory Thanks a lot for working on this.

           private static final String TEST_STS_ENDPOINT = "tst.sts.endpoint";
          

          Could you fix the typo in the string?

          @Before
          public void checkSettings() throws Exception {
             conf = new Configuration();
             assumeNotNull(conf.get(TEST_STS_ENABLED));
             testDir = new Path(TEST_DIR);
          }
          

          Would it be ok to rename this function to setUp() to be consistent to the rest of the code. Should TEST_STS_ENABLED has a boolean value? does it have a default value? One more question regarding this boolean value: should this test be enabled when all S3a tests are enabled? Is there any circumstances that we want to enable either STS or the rest of S3A tests?

          useSessionToken(
              sessionCreds.getAccessKeyId(),
              sessionCreds.getSecretAccessKey(),
              sessionCreds.getSessionToken());
          
          assertTrue("Verifying test directory was made: ", fs.exists(testDir));
          

          Would you put fs.exists(..) and fs.mkdirs() into the same function?

          Last, is marking the functions throwing IOException sufficient, instead of Exception?

          The rest LTGM. I'd give +1 once the above are addressed.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Hi, Sean Mackrory Thanks a lot for working on this. private static final String TEST_STS_ENDPOINT = "tst.sts.endpoint" ; Could you fix the typo in the string? @Before public void checkSettings() throws Exception { conf = new Configuration(); assumeNotNull(conf.get(TEST_STS_ENABLED)); testDir = new Path(TEST_DIR); } Would it be ok to rename this function to setUp() to be consistent to the rest of the code. Should TEST_STS_ENABLED has a boolean value? does it have a default value? One more question regarding this boolean value: should this test be enabled when all S3a tests are enabled? Is there any circumstances that we want to enable either STS or the rest of S3A tests? useSessionToken( sessionCreds.getAccessKeyId(), sessionCreds.getSecretAccessKey(), sessionCreds.getSessionToken()); assertTrue( "Verifying test directory was made: " , fs.exists(testDir)); Would you put fs.exists(..) and fs.mkdirs() into the same function? Last, is marking the functions throwing IOException sufficient, instead of Exception? The rest LTGM. I'd give +1 once the above are addressed.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. maybe make the test dependent purely on the presence of the STS credentials and endpoint.
          1. I think your test you should extend AbstractFSContractTestBase, like, say, AbstractFSContractTestBase, so it picks up some of the setup and config there.
          2. for completeness, the
          3. There are also things in ContractTestUtils to do directory deletion in teardown, as well as lots of assertions & basic operations to create files.
          4. I don't think we need to add any more FS manipulation operations other than creating and listing an entry, do we? That'd just be extra work.
          Show
          stevel@apache.org Steve Loughran added a comment - maybe make the test dependent purely on the presence of the STS credentials and endpoint. I think your test you should extend AbstractFSContractTestBase , like, say, AbstractFSContractTestBase , so it picks up some of the setup and config there. for completeness, the There are also things in ContractTestUtils to do directory deletion in teardown, as well as lots of assertions & basic operations to create files. I don't think we need to add any more FS manipulation operations other than creating and listing an entry, do we? That'd just be extra work.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I believe this patch addresses all the feedback:

          • Fixed the typo in TEST_STS_ENDPOINT
          • Renamed checkSettings to setUp()
          • TEST_STS_ENABLED is no longer a thing: using the file system contract instead
          • Extending AbstractFSContractTestBase, and using ContractTestUtils
          • Throwing IOException specifically, instead of the base Exception

          Reran similar tests to last time, and also tested that the test is skipped if the contract doesn't support this.

          One weird thing is that I just cannot get ContractOptions.ACCEPTS_TEMPORARY_CREDENTIALS in scope on my machine when running tests. It should be in scope, but the only way I can get the tests to pass when they should is by redefining ACCEPTS_TEMPORARY_CREDENTIALS in TestS3ATemporaryCredentials, which shouldn't be necessary. I worked with Aaron Fabbri and he was able to run the tests without the redefinition, so I'm not sure what's going on on my machine...

          Show
          mackrorysd Sean Mackrory added a comment - I believe this patch addresses all the feedback: Fixed the typo in TEST_STS_ENDPOINT Renamed checkSettings to setUp() TEST_STS_ENABLED is no longer a thing: using the file system contract instead Extending AbstractFSContractTestBase, and using ContractTestUtils Throwing IOException specifically, instead of the base Exception Reran similar tests to last time, and also tested that the test is skipped if the contract doesn't support this. One weird thing is that I just cannot get ContractOptions.ACCEPTS_TEMPORARY_CREDENTIALS in scope on my machine when running tests. It should be in scope, but the only way I can get the tests to pass when they should is by redefining ACCEPTS_TEMPORARY_CREDENTIALS in TestS3ATemporaryCredentials, which shouldn't be necessary. I worked with Aaron Fabbri and he was able to run the tests without the redefinition, so I'm not sure what's going on on my machine...
          Hide
          mackrorysd Sean Mackrory added a comment -

          And to be clear, I'm referring to the use of ACCEPTS_TEMPORARY_CREDENTIALS in TestS3ATemporaryCredentials...

          Show
          mackrorysd Sean Mackrory added a comment - And to be clear, I'm referring to the use of ACCEPTS_TEMPORARY_CREDENTIALS in TestS3ATemporaryCredentials...
          Hide
          mackrorysd Sean Mackrory added a comment -

          Turns out it was a simple matter of running tests in hadoop-tools/hadoop-aws when I hadn't rebuilt the parent... So the patch is good - tests consistently pass / skip when they should without further modifications.

          Show
          mackrorysd Sean Mackrory added a comment - Turns out it was a simple matter of running tests in hadoop-tools/hadoop-aws when I hadn't rebuilt the parent... So the patch is good - tests consistently pass / skip when they should without further modifications.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          +1 mvninstall 10m 10s trunk passed
          +1 compile 12m 34s trunk passed with JDK v1.8.0_66
          +1 compile 12m 28s trunk passed with JDK v1.7.0_91
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 2m 18s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          -1 findbugs 0m 14s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 2m 27s trunk passed with JDK v1.8.0_66
          +1 javadoc 2m 11s trunk passed with JDK v1.7.0_91
          +1 mvninstall 2m 42s the patch passed
          +1 compile 13m 15s the patch passed with JDK v1.8.0_66
          +1 javac 13m 15s the patch passed
          +1 compile 11m 55s the patch passed with JDK v1.7.0_91
          +1 javac 11m 55s the patch passed
          +1 checkstyle 1m 17s the patch passed
          +1 mvnsite 1m 58s the patch passed
          +1 mvneclipse 0m 49s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 0m 14s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml)
          +1 javadoc 1m 46s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91
          +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.8.0_66.
          -1 unit 9m 19s hadoop-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 18s hadoop-aws in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_91.
          -1 unit 9m 41s hadoop-common in the patch failed with JDK v1.7.0_91.
          +1 unit 0m 20s hadoop-aws in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 35s Patch does not generate ASF License warnings.
          109m 53s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl
          JDK v1.7.0_91 Failed junit tests hadoop.fs.shell.find.TestFind
            hadoop.ipc.TestIPC
            hadoop.test.TestTimedOutTestsListener



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776874/HADOOP-12537.002.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 8b0796e6e174 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 21daa6c
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: .
          Max memory used 76MB
          Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 10m 10s trunk passed +1 compile 12m 34s trunk passed with JDK v1.8.0_66 +1 compile 12m 28s trunk passed with JDK v1.7.0_91 +1 checkstyle 1m 27s trunk passed +1 mvnsite 2m 18s trunk passed +1 mvneclipse 0m 59s trunk passed -1 findbugs 0m 14s branch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 2m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_91 +1 mvninstall 2m 42s the patch passed +1 compile 13m 15s the patch passed with JDK v1.8.0_66 +1 javac 13m 15s the patch passed +1 compile 11m 55s the patch passed with JDK v1.7.0_91 +1 javac 11m 55s the patch passed +1 checkstyle 1m 17s the patch passed +1 mvnsite 1m 58s the patch passed +1 mvneclipse 0m 49s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 0m 14s patch/hadoop-project no findbugs output file (hadoop-project/target/findbugsXml.xml) +1 javadoc 1m 46s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 50s the patch passed with JDK v1.7.0_91 +1 unit 0m 12s hadoop-project in the patch passed with JDK v1.8.0_66. -1 unit 9m 19s hadoop-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 18s hadoop-aws in the patch passed with JDK v1.8.0_66. +1 unit 0m 11s hadoop-project in the patch passed with JDK v1.7.0_91. -1 unit 9m 41s hadoop-common in the patch failed with JDK v1.7.0_91. +1 unit 0m 20s hadoop-aws in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 109m 53s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.metrics2.impl.TestMetricsSystemImpl JDK v1.7.0_91 Failed junit tests hadoop.fs.shell.find.TestFind   hadoop.ipc.TestIPC   hadoop.test.TestTimedOutTestsListener Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776874/HADOOP-12537.002.patch JIRA Issue HADOOP-12537 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 8b0796e6e174 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21daa6c findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-project hadoop-tools/hadoop-aws U: . Max memory used 76MB Powered by Apache Yetus 0.1.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8220/console This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          The failures on hadoop-common were also occurring in recently committed patches - I do not believe it's related to mine.

          Show
          mackrorysd Sean Mackrory added a comment - The failures on hadoop-common were also occurring in recently committed patches - I do not believe it's related to mine.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Can everyone interested in having this in 2.8 apply and test this patch? thanks

          Show
          stevel@apache.org Steve Loughran added a comment - Can everyone interested in having this in 2.8 apply and test this patch? thanks
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ...still hoping that some of the watchers can play with this; I'm looking at some of the others

          Show
          stevel@apache.org Steve Loughran added a comment - ...still hoping that some of the watchers can play with this; I'm looking at some of the others
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-12537 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776874/HADOOP-12537.002.patch
          JIRA Issue HADOOP-12537
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9012/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-12537 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776874/HADOOP-12537.002.patch JIRA Issue HADOOP-12537 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9012/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Linking to HADOOP-13078, reverting the 2.8+ AWS SDK to the full one. do that and then this patch doesn't need any new JAR, isn't going to cause problems if the STS jar isn't on the classpath...so there's no risk JAR-wise from adding this patch

          Show
          stevel@apache.org Steve Loughran added a comment - Linking to HADOOP-13078 , reverting the 2.8+ AWS SDK to the full one. do that and then this patch doesn't need any new JAR, isn't going to cause problems if the STS jar isn't on the classpath...so there's no risk JAR-wise from adding this patch
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm marking this as a dependency of HADOOP-12723, which will allow any named provider to be loaded on the classpath.

          This will then allow STS to be supported if the relevant JAR is on the CP, and a (to be written) credential provider does the binding.

          Would you be able to move this patch to work with that...a whole new provider which took the proposed config option and set things up. The new JAR dependency would be declared "provided", so it'd be there for build and test, but not required downstream unless the feature was required

          Show
          stevel@apache.org Steve Loughran added a comment - I'm marking this as a dependency of HADOOP-12723 , which will allow any named provider to be loaded on the classpath. This will then allow STS to be supported if the relevant JAR is on the CP, and a (to be written) credential provider does the binding. Would you be able to move this patch to work with that...a whole new provider which took the proposed config option and set things up. The new JAR dependency would be declared "provided", so it'd be there for build and test, but not required downstream unless the feature was required
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Sean Mackrory -I've committed HADOOP-12723, which will allow STS support to be an optional auth provider.

          Would you be able to move this patch/feature to that plugin mechanism?

          Show
          stevel@apache.org Steve Loughran added a comment - Sean Mackrory -I've committed HADOOP-12723 , which will allow STS support to be an optional auth provider. Would you be able to move this patch/feature to that plugin mechanism?
          Hide
          mackrorysd Sean Mackrory added a comment -

          Yeah thanks for the update - I'l rewrite the patch shortly...

          Show
          mackrorysd Sean Mackrory added a comment - Yeah thanks for the update - I'l rewrite the patch shortly...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Looking forward to it.

          I should warn that the patch will need a paragraph in the index.md on authing with session credentials, and in the testing section at the bottom there, how to test.

          Test-wise explaining how to turn on the feature before a full test run seems about the only way to reasonably do it.

          Show
          stevel@apache.org Steve Loughran added a comment - Looking forward to it. I should warn that the patch will need a paragraph in the index.md on authing with session credentials, and in the testing section at the bottom there, how to test. Test-wise explaining how to turn on the feature before a full test run seems about the only way to reasonably do it.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Attaching an updated patch. Most (but not all) of the regions seem to be failing if the bucket and STS endpoint are for different regions. I'm having additional issues with Frankfurt and Seoul I still need to look into.

          Show
          mackrorysd Sean Mackrory added a comment - Attaching an updated patch. Most (but not all) of the regions seem to be failing if the bucket and STS endpoint are for different regions. I'm having additional issues with Frankfurt and Seoul I still need to look into.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Forgot to git add the new files...

          Show
          mackrorysd Sean Mackrory added a comment - Forgot to git add the new files...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          interesting: frankfurt is rejecting me with a 400, even if I use normal credentials.

          Does moving to a later AWS library make a difference?

          Show
          stevel@apache.org Steve Loughran added a comment - interesting: frankfurt is rejecting me with a 400, even if I use normal credentials. Does moving to a later AWS library make a difference?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 14s trunk passed
          +1 compile 6m 32s trunk passed
          +1 checkstyle 1m 19s trunk passed
          +1 mvnsite 1m 20s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 6m 5s the patch passed
          +1 javac 6m 5s the patch passed
          -1 checkstyle 1m 19s root: The patch generated 8 new + 22 unchanged - 0 fixed = 30 total (was 22)
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 1m 14s the patch passed
          +1 unit 0m 9s hadoop-project in the patch passed.
          +1 unit 7m 38s hadoop-common in the patch passed.
          +1 unit 0m 13s hadoop-aws in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          41m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805726/HADOOP-12537.004.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 43d479c8bffb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ac95448
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 14s trunk passed +1 compile 6m 32s trunk passed +1 checkstyle 1m 19s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 35s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 37s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 6m 5s the patch passed +1 javac 6m 5s the patch passed -1 checkstyle 1m 19s root: The patch generated 8 new + 22 unchanged - 0 fixed = 30 total (was 22) +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 51s the patch passed +1 javadoc 1m 14s the patch passed +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 7m 38s hadoop-common in the patch passed. +1 unit 0m 13s hadoop-aws in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 41m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805726/HADOOP-12537.004.patch JIRA Issue HADOOP-12537 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 43d479c8bffb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac95448 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9561/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I'm getting a 301 Moved Permanently as though my bucket and S3 endpoint weren't matching up, but I've double-checked they're right. I also just tested with version 1.11.2 of the SDK and did not have success. I'm less concerned by the fact that it seems to require matching STS and S3 regions much of the time, although it sounds from my previous comments like it was more flexible last time around.

          Show
          mackrorysd Sean Mackrory added a comment - I'm getting a 301 Moved Permanently as though my bucket and S3 endpoint weren't matching up, but I've double-checked they're right. I also just tested with version 1.11.2 of the SDK and did not have success. I'm less concerned by the fact that it seems to require matching STS and S3 regions much of the time, although it sounds from my previous comments like it was more flexible last time around.
          Show
          stevel@apache.org Steve Loughran added a comment - Frankfurt is AWSv4 signatures only: http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html this is why it has a bit of a reputation http://stackoverflow.com/questions/33828588/spark-cannot-read-files-stored-on-aws-s3-in-frankfurt-region-ireland-region-wor https://community.cloudera.com/t5/Storage-Random-Access-HDFS/cloudera-does-not-support-access-to-s3-within-eu-frankfurt-Aws/td-p/32369
          Hide
          mackrorysd Sean Mackrory added a comment -

          Okay I figured it out - I had forgotten about contract-test-options.xml and I still had a different bucket configured there, so it wasn't consistently using a bucket in the region I thought. The test has now passed for me in all regions, and it would appear you can use whatever STS endpoint you want, and Frankfurt and Seoul are just a bit more picky about bucket location.

          Show
          mackrorysd Sean Mackrory added a comment - Okay I figured it out - I had forgotten about contract-test-options.xml and I still had a different bucket configured there, so it wasn't consistently using a bucket in the region I thought. The test has now passed for me in all regions, and it would appear you can use whatever STS endpoint you want, and Frankfurt and Seoul are just a bit more picky about bucket location.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This patch 005; patch 004 merged in and reworked somewhat

          Production:

          1. throwing an explicit CredentialInitializationException (backported to the other providers). This declares itself as non-retryable, and, being an explicit type, can be looked for in names.
          2. Changed the exception text message.

          Test

          1. moved the STS option flag out of the contract tests, instead adding one to the S3A tests only.
          2. added a test for missing session token.
          3. extended the existing test with an attempt to create an FS with an invalid token.

          Docs

          1. more detail, an example, and some test docs.

          The new test —taking the newly issued session token and trying to init with a now-invalid triple of (key-id, key-secret, sessionId) is something I'd have expected to fail. It isn't. This is something that needs to be fixed.

          Hypotheses

          • I've misunderstood something
          • credential setup isn't working as expected; perhaps the permanent keys are being picked up, not these new ones.
          • AWS is doing something underneath.

          Sean Mackrory: could you look at that? Once that test is passing I think we're pretty much good to go here

          Show
          stevel@apache.org Steve Loughran added a comment - This patch 005; patch 004 merged in and reworked somewhat Production: throwing an explicit CredentialInitializationException (backported to the other providers). This declares itself as non-retryable, and, being an explicit type, can be looked for in names. Changed the exception text message. Test moved the STS option flag out of the contract tests, instead adding one to the S3A tests only. added a test for missing session token. extended the existing test with an attempt to create an FS with an invalid token. Docs more detail, an example, and some test docs. The new test —taking the newly issued session token and trying to init with a now-invalid triple of (key-id, key-secret, sessionId) is something I'd have expected to fail. It isn't. This is something that needs to be fixed. Hypotheses I've misunderstood something credential setup isn't working as expected; perhaps the permanent keys are being picked up, not these new ones. AWS is doing something underneath. Sean Mackrory : could you look at that? Once that test is passing I think we're pretty much good to go here
          Hide
          stevel@apache.org Steve Loughran added a comment -

          submitting against branch-2

          Show
          stevel@apache.org Steve Loughran added a comment - submitting against branch-2
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 33s branch-2 passed
          +1 compile 6m 3s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 33s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 34s branch-2 passed
          +1 mvnsite 1m 45s branch-2 passed
          +1 mvneclipse 0m 38s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 8s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 1m 3s the patch passed
          +1 compile 5m 41s the patch passed with JDK v1.8.0_91
          +1 javac 5m 41s the patch passed
          +1 compile 6m 21s the patch passed with JDK v1.7.0_101
          +1 javac 6m 21s the patch passed
          -1 checkstyle 1m 20s root: The patch generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9)
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 30s the patch passed
          +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 29s the patch passed with JDK v1.7.0_101
          +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91.
          +1 unit 10m 45s hadoop-common in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101.
          +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          72m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807500/HADOOP-12537-branch-2.005.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux d6ee51086b37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / b29bf5d
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 33s branch-2 passed +1 compile 6m 3s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 33s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 34s branch-2 passed +1 mvnsite 1m 45s branch-2 passed +1 mvneclipse 0m 38s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 8s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 32s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 5m 41s the patch passed with JDK v1.8.0_91 +1 javac 5m 41s the patch passed +1 compile 6m 21s the patch passed with JDK v1.7.0_101 +1 javac 6m 21s the patch passed -1 checkstyle 1m 20s root: The patch generated 2 new + 9 unchanged - 0 fixed = 11 total (was 9) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 30s the patch passed +1 javadoc 1m 16s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 29s the patch passed with JDK v1.7.0_101 +1 unit 0m 8s hadoop-project in the patch passed with JDK v1.8.0_91. +1 unit 10m 45s hadoop-common in the patch passed with JDK v1.8.0_91. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101. +1 unit 8m 28s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 72m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807500/HADOOP-12537-branch-2.005.patch JIRA Issue HADOOP-12537 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux d6ee51086b37 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / b29bf5d Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9639/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          It looks to me like you can append to the session token and it will still work - it must just be parsing what it expects and not looking at the rest. I can make the test fail by prepending, or simply setting the empty string as the session token. As soon as I append to an otherwise valid session token, the test fails because the S3 access still works.

          I thought I also observed behavior where authorization seemed to be cached for a short period of time, but I have been misinterpreting the behavior I just described. Going to run through a few more tweaks of the test case to ensure this is not the case.

          Show
          mackrorysd Sean Mackrory added a comment - It looks to me like you can append to the session token and it will still work - it must just be parsing what it expects and not looking at the rest. I can make the test fail by prepending, or simply setting the empty string as the session token. As soon as I append to an otherwise valid session token, the test fails because the S3 access still works. I thought I also observed behavior where authorization seemed to be cached for a short period of time, but I have been misinterpreting the behavior I just described. Going to run through a few more tweaks of the test case to ensure this is not the case.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ahh, you worked out. Interesting though. How about have the test just set it to a random string and expect a failure there.

          If you do an updated test patch with this in, it'd be great

          Show
          stevel@apache.org Steve Loughran added a comment - ahh, you worked out. Interesting though. How about have the test just set it to a random string and expect a failure there. If you do an updated test patch with this in, it'd be great
          Hide
          mackrorysd Sean Mackrory added a comment -

          Updating the patch to prepend to the existing session token instead (and added a note about how appending doesn't seem to invalidate it), and updating the exception type for what actually gets thrown first when creating the filesystem.

          I'm satisfied there's no 'caching' of the authorization going on like I suspected earlier.

          Show
          mackrorysd Sean Mackrory added a comment - Updating the patch to prepend to the existing session token instead (and added a note about how appending doesn't seem to invalidate it), and updating the exception type for what actually gets thrown first when creating the filesystem. I'm satisfied there's no 'caching' of the authorization going on like I suspected earlier.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 8m 12s trunk passed
          +1 compile 8m 1s trunk passed
          +1 checkstyle 1m 27s trunk passed
          +1 mvnsite 1m 42s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 5s trunk passed
          +1 javadoc 1m 40s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 8m 37s the patch passed
          +1 javac 8m 37s the patch passed
          -1 checkstyle 1m 28s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 23s the patch passed
          +1 javadoc 1m 23s the patch passed
          +1 unit 0m 7s hadoop-project in the patch passed.
          -1 unit 9m 8s hadoop-common in the patch failed.
          +1 unit 0m 16s hadoop-aws in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          52m 52s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807725/HADOOP-12537.006.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 2469379c0102 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / aadb77e
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 8m 12s trunk passed +1 compile 8m 1s trunk passed +1 checkstyle 1m 27s trunk passed +1 mvnsite 1m 42s trunk passed +1 mvneclipse 0m 41s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 5s trunk passed +1 javadoc 1m 40s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 8m 37s the patch passed +1 javac 8m 37s the patch passed -1 checkstyle 1m 28s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 23s the patch passed +1 javadoc 1m 23s the patch passed +1 unit 0m 7s hadoop-project in the patch passed. -1 unit 9m 8s hadoop-common in the patch failed. +1 unit 0m 16s hadoop-aws in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 52m 52s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807725/HADOOP-12537.006.patch JIRA Issue HADOOP-12537 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 2469379c0102 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aadb77e Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9647/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          checkstyle

          ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:22:import java.nio.file.AccessDeniedException;:8: Unused import - java.nio.file.AccessDeniedException.
          ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:82:  public void TestSTS() throws IOException {:15: Name 'TestSTS' must match pattern '^[a-z][a-zA-Z0-9]*$'.
          
          Show
          stevel@apache.org Steve Loughran added a comment - checkstyle ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:22: import java.nio.file.AccessDeniedException;:8: Unused import - java.nio.file.AccessDeniedException. ./hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java:82: public void TestSTS() throws IOException {:15: Name 'TestSTS' must match pattern '^[a-z][a-zA-Z0-9]*$'.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-12537: patch 007: 006 + checkstyle fixes

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-12537 : patch 007: 006 + checkstyle fixes
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 40s branch-2 passed
          +1 compile 6m 15s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 34s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 29s branch-2 passed
          +1 mvnsite 1m 28s branch-2 passed
          +1 mvneclipse 0m 35s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 7s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 29s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 3s the patch passed
          +1 compile 6m 16s the patch passed with JDK v1.8.0_91
          +1 javac 6m 16s the patch passed
          +1 compile 6m 32s the patch passed with JDK v1.7.0_101
          +1 javac 6m 32s the patch passed
          -1 checkstyle 1m 23s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7)
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 32s the patch passed
          +1 javadoc 1m 23s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 30s the patch passed with JDK v1.7.0_101
          +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101.
          +1 unit 8m 39s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          70m 37s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.net.TestClusterTopology
            hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809198/HADOOP-12537-branch-2.007.patch
          JIRA Issue HADOOP-12537
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c97c11c00975 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / aef6e45
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/testReport/
          modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 40s branch-2 passed +1 compile 6m 15s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 34s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 29s branch-2 passed +1 mvnsite 1m 28s branch-2 passed +1 mvneclipse 0m 35s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 7s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 29s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 6m 16s the patch passed with JDK v1.8.0_91 +1 javac 6m 16s the patch passed +1 compile 6m 32s the patch passed with JDK v1.7.0_101 +1 javac 6m 32s the patch passed -1 checkstyle 1m 23s root: The patch generated 2 new + 7 unchanged - 0 fixed = 9 total (was 7) +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 32s the patch passed +1 javadoc 1m 23s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 30s the patch passed with JDK v1.7.0_101 +1 unit 0m 9s hadoop-project in the patch passed with JDK v1.7.0_101. +1 unit 8m 39s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 13s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 70m 37s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.net.TestClusterTopology   hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12809198/HADOOP-12537-branch-2.007.patch JIRA Issue HADOOP-12537 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c97c11c00975 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / aef6e45 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/testReport/ modules C: hadoop-project hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9708/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1; I'll fix the second checkstyle in the commit

          Show
          stevel@apache.org Steve Loughran added a comment - +1; I'll fix the second checkstyle in the commit
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Thanks for doing this Sean —a lot of people will appreciate it. Changing the title slightly for the generated release notes.

          Show
          stevel@apache.org Steve Loughran added a comment - Thanks for doing this Sean —a lot of people will appreciate it. Changing the title slightly for the generated release notes.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9938 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9938/)
          HADOOP-12537 S3A to support Amazon STS temporary credentials. (stevel: rev 31ffaf76f2b6e1fd2a141daa4daaebdfecefe727)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/CredentialInitializationException.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9938 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9938/ ) HADOOP-12537 S3A to support Amazon STS temporary credentials. (stevel: rev 31ffaf76f2b6e1fd2a141daa4daaebdfecefe727) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/BasicAWSCredentialsProvider.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java hadoop-tools/hadoop-aws/pom.xml hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/CredentialInitializationException.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-project/pom.xml
          Hide
          mackrorysd Sean Mackrory added a comment -

          Thanks - looks good to me!

          Show
          mackrorysd Sean Mackrory added a comment - Thanks - looks good to me!

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              mackrorysd Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development