Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15650

Add custom InstanceProfileCredentialsProvider with more resilience to throttling

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Add our own InstanceProfileCredentialsProvider class which uses the AWS implementation to retrieve credentials from EC2's instance info, but more resilient to overloading.

      1. pass in client config with retry logic (HADOOP-15603)
      2. use Invoke.retry() to retry
      3. log/measure failures
      4. maybe use the Async feature of the AWS SDK class, so that credential renewer doesn't block IO.
      5. be shared amongst all AWS auth chains which need these credentials.

      The singleton we current use for IAM auth doesn't do async, which is good as it ensures that we don't prematurely close it when AWSCredentialProviderList.close() closes its children.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: