Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13389

TestS3ATemporaryCredentials.testSTS error when using IAM credentials

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      org.apache.hadoop.fs.s3a.TestS3ATemporaryCredentials.testSTS throws a 403 AccessDenied when run without any AWS credentials (access key and secret key) in the config.

      com.amazonaws.AmazonServiceException: Cannot call GetSessionToken with session credentials (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: XXXXX)
      	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
      	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
      	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
      	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
      	at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1106)
      	at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.getSessionToken(AWSSecurityTokenServiceClient.java:355)
      	at org.apache.hadoop.fs.s3a.TestS3ATemporaryCredentials.testSTS(TestS3ATemporaryCredentials.java:105)
      

      It fails because the InstanceProfileCredentialsProvider in the credentials chain (on line 91) is used, but an instance profile always provides a temporary credential and GetSessionToken requires a long-term (not temporary) credential.

      Suggestion on how to fix this test case?

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          Hello Steven K. Wong. All of the hadoop-aws tests should be getting skipped if there are no AWS credentials configured. That's accomplished via this code in pom.xml:

              <profile>
                <id>tests-off</id>
                <activation>
                  <file>
                    <missing>src/test/resources/auth-keys.xml</missing>
                  </file>
                </activation>
                <properties>
                  <maven.test.skip>true</maven.test.skip>
                </properties>
              </profile>
          

          Is there something unique in your environment that is causing this test to run even when credentials are not configured?

          Show
          cnauroth Chris Nauroth added a comment - Hello Steven K. Wong . All of the hadoop-aws tests should be getting skipped if there are no AWS credentials configured. That's accomplished via this code in pom.xml: <profile> <id>tests-off</id> <activation> <file> <missing>src/test/resources/auth-keys.xml</missing> </file> </activation> <properties> <maven.test.skip> true </maven.test.skip> </properties> </profile> Is there something unique in your environment that is causing this test to run even when credentials are not configured?
          Hide
          slider Steven K. Wong added a comment - - edited

          I have auth-keys.xml (that only configures test.fs.s3a.name), because I intend to run the S3A tests. All S3A tests – except TestS3ATemporaryCredentials.testSTS – succeed for me.

          The InstanceProfileCredentialsProvider object on line 93 is unhelpful because its temporary credential is not compatible with the getSessionToken call on line 105 (as explained above). Hence, at a minimum I think InstanceProfileCredentialsProvider should be removed from the credentials chain in the test case. But that doesn't fix the test case failure. Perhaps testSTS should explicitly check for the absence of credentials in the config and skip itself (like what line 83 does)?

          Show
          slider Steven K. Wong added a comment - - edited I have auth-keys.xml (that only configures test.fs.s3a.name), because I intend to run the S3A tests. All S3A tests – except TestS3ATemporaryCredentials.testSTS – succeed for me. The InstanceProfileCredentialsProvider object on line 93 is unhelpful because its temporary credential is not compatible with the getSessionToken call on line 105 (as explained above). Hence, at a minimum I think InstanceProfileCredentialsProvider should be removed from the credentials chain in the test case. But that doesn't fix the test case failure. Perhaps testSTS should explicitly check for the absence of credentials in the config and skip itself (like what line 83 does)?
          Hide
          cnauroth Chris Nauroth added a comment -

          Steven K. Wong, thank you for the further details. I think I understand now. It sounds like you are trying to run the S3A test suite without an AWS access key ID and secret access key, instead relying on instance profile credentials provided in an EC2 VM.

          The simplest immediate workaround for you is likely to set the following in your auth-keys.xml file:

                  <property>
                    <name>test.fs.s3a.sts.enabled</name>
                    <value>false</value>
                  </property>
          

          However, I also agree that if the instance profile credentials are never suitable for this test case, then we would do well to remove InstanceProfileCredentialsProvider from the test and add explicit detection to skip if there is no access key ID and secret access key. S3AUtils#getAWSAccessKeys and S3xLoginHelper class are likely to be helpful for that logic.

          Show
          cnauroth Chris Nauroth added a comment - Steven K. Wong , thank you for the further details. I think I understand now. It sounds like you are trying to run the S3A test suite without an AWS access key ID and secret access key, instead relying on instance profile credentials provided in an EC2 VM. The simplest immediate workaround for you is likely to set the following in your auth-keys.xml file: <property> <name>test.fs.s3a.sts.enabled</name> <value> false </value> </property> However, I also agree that if the instance profile credentials are never suitable for this test case, then we would do well to remove InstanceProfileCredentialsProvider from the test and add explicit detection to skip if there is no access key ID and secret access key. S3AUtils#getAWSAccessKeys and S3xLoginHelper class are likely to be helpful for that logic.
          Hide
          slider Steven K. Wong added a comment -

          Chris Nauroth, thanks for the suggestions. I'll work on a patch.

          Show
          slider Steven K. Wong added a comment - Chris Nauroth , thanks for the suggestions. I'll work on a patch.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steven K. Wong, thank you for volunteering to write the patch. I have assigned the issue to you.

          Show
          cnauroth Chris Nauroth added a comment - Steven K. Wong , thank you for volunteering to write the patch. I have assigned the issue to you.
          Hide
          slider Steven K. Wong added a comment -

          Attaching HADOOP-13389.001.patch – tested in EC2 in us-east-1 with S3 in US Standard.

          Show
          slider Steven K. Wong added a comment - Attaching HADOOP-13389 .001.patch – tested in EC2 in us-east-1 with S3 in US Standard.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 42s trunk passed
          +1 compile 0m 15s trunk passed
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 mvneclipse 1m 8s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 15s the patch passed
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 18s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 0m 13s hadoop-aws in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          14m 24s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819189/HADOOP-13389.001.patch
          JIRA Issue HADOOP-13389
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 2e59c17d7c48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 38128ba
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10040/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10040/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 42s trunk passed +1 compile 0m 15s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 1m 8s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 0m 13s hadoop-aws in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 14m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12819189/HADOOP-13389.001.patch JIRA Issue HADOOP-13389 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e59c17d7c48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 38128ba Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10040/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10040/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          1, applied to branch-2.8

          I didn't retest with temporary credentials; I did do a regression test run against S3 ireland using my normal creds...that verifies that the patch hasn't broken that code path.

          Show
          stevel@apache.org Steve Loughran added a comment - 1, applied to branch-2.8 I didn't retest with temporary credentials; I did do a regression test run against S3 ireland using my normal creds...that verifies that the patch hasn't broken that code path.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10142 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10142/)
          HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM (stevel: rev 7052ca8804b2c69022cf7cfc6b95f21aee3c3640)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10142 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10142/ ) HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM (stevel: rev 7052ca8804b2c69022cf7cfc6b95f21aee3c3640) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ATemporaryCredentials.java

            People

            • Assignee:
              slider Steven K. Wong
              Reporter:
              slider Steven K. Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development