Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13727

S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider.

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None

      Description

      When running in an EC2 VM, S3A can make use of InstanceProfileCredentialsProvider from the AWS SDK to obtain credentials from the EC2 Instance Metadata Service. We have observed that for a highly multi-threaded application, this may generate a high number of calls to the Instance Metadata Service. The service may throttle the client by replying with an HTTP 429 response or forcibly closing connections. We can greatly reduce the number of calls to the service by enforcing that all threads use a single shared instance of InstanceProfileCredentialsProvider.

        Attachments

        1. HADOOP-13727-branch-2.005.patch
          48 kB
          Chris Nauroth
        2. HADOOP-13727-branch-2.004.patch
          48 kB
          Chris Nauroth
        3. HADOOP-13727-branch-2.003.patch
          48 kB
          Chris Nauroth
        4. HADOOP-13727-branch-2.002.patch
          37 kB
          Chris Nauroth
        5. HADOOP-13727-branch-2.001.patch
          36 kB
          Chris Nauroth

          Issue Links

            Activity

              People

              • Assignee:
                cnauroth Chris Nauroth
                Reporter:
                rajesh.balamohan Rajesh Balamohan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: