Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12807

S3AFileSystem should read AWS credentials from environment variables

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Release Note:
      Adds support to S3AFileSystem for reading AWS credentials from environment variables.

      Description

      Unlike the DefaultAWSCredentialsProviderChain in the AWS SDK, the AWSCredentialsProviderChain constructed by S3AFileSystem does not include an EnvironmentVariableCredentialsProvider instance. This prevents users from supplying AWS credentials in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, which is the only alternative in some scenarios.

      In my scenario, I need to access S3 from within a test running in a CI environment that does not support IAM roles but does allow me to supply encrypted environment variables. Thus, the only secure approach I can use is to supply my AWS credentials in environment variables (plaintext configuration files are out of the question).

      1. HADOOP-12807-1.patch
        1 kB
        Tobin Baker
      2. HADOOP-12807-branch-2-004.patch
        3 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-12548 should address your needs: linking to it. Tobin —can you download/apply that patch and see if you can add the provider you need?

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-12548 should address your needs: linking to it. Tobin —can you download/apply that patch and see if you can add the provider you need?
          Hide
          tdbaker Tobin Baker added a comment -

          Thanks for the response, Steve. I did see that patch before I opened this JIRA, but I felt this issue was worth addressing separately for the following reasons.

          All of the AWS SDKs, and most standard AWS tools (including the AWS CLI) accept AWS credentials from the standard environment variables AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. It's what any experienced user of these tools would expect from S3A, and it's a very useful capability (as in the CI scenario I described). I would say it's at least as useful and important as the ability to accept credentials from an IAM role (which S3A also supports out-of-the-box). Given that it's literally a one-line change (see attached patch), that all standard AWS tools support the same functionality by default, and that it enables common use cases that can't be addressed otherwise, I think S3A would need a very good reason not to support EnvironmentVariableCredentialsProvider by default. If there are valid security concerns with this change, then I certainly would like to hear about them. Environment variables can certainly be used insecurely, but they can hardly be any worse than the widely supported practice (in Hadoop and elsewhere) of storing credentials in plaintext configuration files.

          So, given that environment variable support is trivial to implement, extremely low-risk, widely useful, and is already supported by pretty much all standard AWS tools, I thought it would be reasonable to support it implicitly rather than force clients to implement an extension.

          Show
          tdbaker Tobin Baker added a comment - Thanks for the response, Steve. I did see that patch before I opened this JIRA, but I felt this issue was worth addressing separately for the following reasons. All of the AWS SDKs, and most standard AWS tools (including the AWS CLI) accept AWS credentials from the standard environment variables AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY . It's what any experienced user of these tools would expect from S3A, and it's a very useful capability (as in the CI scenario I described). I would say it's at least as useful and important as the ability to accept credentials from an IAM role (which S3A also supports out-of-the-box). Given that it's literally a one-line change (see attached patch), that all standard AWS tools support the same functionality by default, and that it enables common use cases that can't be addressed otherwise, I think S3A would need a very good reason not to support EnvironmentVariableCredentialsProvider by default. If there are valid security concerns with this change, then I certainly would like to hear about them. Environment variables can certainly be used insecurely, but they can hardly be any worse than the widely supported practice (in Hadoop and elsewhere) of storing credentials in plaintext configuration files. So, given that environment variable support is trivial to implement, extremely low-risk, widely useful, and is already supported by pretty much all standard AWS tools, I thought it would be reasonable to support it implicitly rather than force clients to implement an extension.
          Hide
          tdbaker Tobin Baker added a comment -

          Patch is attached. Tested against both trunk and the 2.7.2 release. Please let me know if I need to add a unit test (I omitted one because I'm simply using the existing EnvironmentVariableCredentialsProvider).

          Show
          tdbaker Tobin Baker added a comment - Patch is attached. Tested against both trunk and the 2.7.2 release. Please let me know if I need to add a unit test (I omitted one because I'm simply using the existing EnvironmentVariableCredentialsProvider ).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 9s trunk passed
          +1 compile 0m 11s trunk passed with JDK v1.8.0_72
          +1 compile 0m 14s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 19s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.8.0_72
          +1 javac 0m 10s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.7.0_95
          +1 javac 0m 12s the patch passed
          -1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 1 new + 40 unchanged - 1 fixed = 41 total (was 41)
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_72
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
          +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_72.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          13m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788166/HADOOP-12807-1.patch
          JIRA Issue HADOOP-12807
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f082af0767db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 77ba5ad
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 9s trunk passed +1 compile 0m 11s trunk passed with JDK v1.8.0_72 +1 compile 0m 14s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 14s the patch passed +1 compile 0m 10s the patch passed with JDK v1.8.0_72 +1 javac 0m 10s the patch passed +1 compile 0m 12s the patch passed with JDK v1.7.0_95 +1 javac 0m 12s the patch passed -1 checkstyle 0m 12s hadoop-tools/hadoop-aws: patch generated 1 new + 40 unchanged - 1 fixed = 41 total (was 41) +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 0m 10s hadoop-aws in the patch passed with JDK v1.8.0_72. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 13m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788166/HADOOP-12807-1.patch JIRA Issue HADOOP-12807 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f082af0767db 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77ba5ad Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8639/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I now understand; — and I see it looks like a simple change. But while It may just be a one-liner, regression testing means that a one line change has to be treated as seriously as any other, and added to the release process.

          And it's going to be hard to test as we can't set env vars inside the test JVM. You are going to have to

          1. document the test process in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          2. follow the contribution instructions in https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure

          It's not going to get a look at until that process is followed. Jenkins doesn't test against s3; the extra requirements are there to make sure that whoever submits it has at least run and passed the tests.

          target release would probably be 2.9 + possible backports; please leave out the "fix-version" until the fix is applied, as that confuses the release notes. Thanks.

          Show
          stevel@apache.org Steve Loughran added a comment - I now understand; — and I see it looks like a simple change. But while It may just be a one-liner, regression testing means that a one line change has to be treated as seriously as any other, and added to the release process. And it's going to be hard to test as we can't set env vars inside the test JVM. You are going to have to document the test process in hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md follow the contribution instructions in https://wiki.apache.org/hadoop/HowToContribute#Submitting_patches_against_object_stores_such_as_Amazon_S3.2C_OpenStack_Swift_and_Microsoft_Azure It's not going to get a look at until that process is followed. Jenkins doesn't test against s3; the extra requirements are there to make sure that whoever submits it has at least run and passed the tests. target release would probably be 2.9 + possible backports; please leave out the "fix-version" until the fix is applied, as that confuses the release notes. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-12807 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788166/HADOOP-12807-1.patch
          JIRA Issue HADOOP-12807
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9663/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-12807 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12788166/HADOOP-12807-1.patch JIRA Issue HADOOP-12807 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9663/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002: this is patch 001 updated to apply against the current code

          +added a quick paragraph to the docs, and highlit that the env vars don't propagate to submitted jobs. Don't want to field support calls on that.

          If Yetus is happy with this I'll commit it.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002: this is patch 001 updated to apply against the current code +added a quick paragraph to the docs, and highlit that the env vars don't propagate to submitted jobs. Don't want to field support calls on that. If Yetus is happy with this I'll commit it.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HADOOP-12807 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808191/HADOOP-12807-branch-2-002.patch
          JIRA Issue HADOOP-12807
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9664/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HADOOP-12807 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808191/HADOOP-12807-branch-2-002.patch JIRA Issue HADOOP-12807 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9664/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          don't know why the patch didn't take. Trying again, rebased to branch-2 again

          Show
          stevel@apache.org Steve Loughran added a comment - don't know why the patch didn't take. Trying again, rebased to branch-2 again
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 8s HADOOP-12807 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808413/HADOOP-12807-branch-2-002.patch
          JIRA Issue HADOOP-12807
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9666/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 8s HADOOP-12807 does not apply to branch-2. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808413/HADOOP-12807-branch-2-002.patch JIRA Issue HADOOP-12807 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9666/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 004; 3KB long, This had better apply to branch-2 or I'm in a mess

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 004; 3KB long, This had better apply to branch-2 or I'm in a mess
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 45s branch-2 passed
          +1 compile 0m 12s branch-2 passed with JDK v1.8.0_91
          +1 compile 0m 13s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 0m 15s branch-2 passed
          +1 mvnsite 0m 20s branch-2 passed
          +1 mvneclipse 0m 14s branch-2 passed
          +1 findbugs 0m 40s branch-2 passed
          +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 11s the patch passed with JDK v1.8.0_91
          +1 javac 0m 11s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.7.0_101
          +1 javac 0m 12s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          15m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808463/HADOOP-12807-branch-2-004.patch
          JIRA Issue HADOOP-12807
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 45ad7bceeacf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 074588d
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9668/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9668/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 45s branch-2 passed +1 compile 0m 12s branch-2 passed with JDK v1.8.0_91 +1 compile 0m 13s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 0m 15s branch-2 passed +1 mvnsite 0m 20s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 findbugs 0m 40s branch-2 passed +1 javadoc 0m 16s branch-2 passed with JDK v1.8.0_91 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 11s the patch passed with JDK v1.8.0_91 +1 javac 0m 11s the patch passed +1 compile 0m 12s the patch passed with JDK v1.7.0_101 +1 javac 0m 12s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_101 +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 15m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808463/HADOOP-12807-branch-2-004.patch JIRA Issue HADOOP-12807 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 45ad7bceeacf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 074588d Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9668/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9668/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Yetus is happy apart from the tests.

          There aren't tests as there isn't an easy way to set up an execution environment with env vars for one test suite, and not for the others. I've manually verified it by doing a full hadoop release and performing `hadoop fs` commands.

          +1

          Show
          stevel@apache.org Steve Loughran added a comment - Yetus is happy apart from the tests. There aren't tests as there isn't an easy way to set up an execution environment with env vars for one test suite, and not for the others. I've manually verified it by doing a full hadoop release and performing `hadoop fs` commands. +1
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Thanks: patch applied to 2.8

          Tobin, I hope you aren't letting anyone untrusted submit patches to that CI system? As if they can print your env vars, they get your secrets.

          Given that the env vars supported include transient session tokens, you may be able to get away with session tokens there; it may mean that the STS SDK JAR needs to go on to the CP. If you do try this —let us know how you get on.

          Show
          stevel@apache.org Steve Loughran added a comment - Thanks: patch applied to 2.8 Tobin, I hope you aren't letting anyone untrusted submit patches to that CI system? As if they can print your env vars, they get your secrets. Given that the env vars supported include transient session tokens, you may be able to get away with session tokens there; it may mean that the STS SDK JAR needs to go on to the CP. If you do try this —let us know how you get on.
          Hide
          tdbaker Tobin Baker added a comment -

          Thanks so much for getting this in, and sorry for slacking off on the tests!

          I don't think our CI configuration exposes us to much risk since the IAM user whose env vars are encrypted in our .travis.yml file has no permissions except read/write access to a dedicated test data S3 bucket. Also, our Travis account is restricted to users with write privileges on our Github repo, which is confined to our team.

          Show
          tdbaker Tobin Baker added a comment - Thanks so much for getting this in, and sorry for slacking off on the tests! I don't think our CI configuration exposes us to much risk since the IAM user whose env vars are encrypted in our .travis.yml file has no permissions except read/write access to a dedicated test data S3 bucket. Also, our Travis account is restricted to users with write privileges on our Github repo, which is confined to our team.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9915 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9915/)
          HADOOP-12807 S3AFileSystem should read AWS credentials from environment (stevel: rev a3f78d8fa83f07f9183f3546203a191fcf50008c)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9915 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9915/ ) HADOOP-12807 S3AFileSystem should read AWS credentials from environment (stevel: rev a3f78d8fa83f07f9183f3546203a191fcf50008c) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

            People

            • Assignee:
              tdbaker Tobin Baker
              Reporter:
              tdbaker Tobin Baker
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development