Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-14237

S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      EC2, AWS

    • Target Version/s:

      Description

      When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails getting the instance profile credentials, eventually all jobs on the cluster fail. Since a number of S3A clients (all mappers and reducers) try to get the credentials, the AWS credential endpoint starts responding 5xx and 4xx error codes.

      SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, but it still does not share the credentials with other EC2 nodes / JVM processes.

      This issue prevents users from creating Hadoop clusters on EC2

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kazuyukitanimura Kazuyuki Tanimura
                Reporter:
                kazuyukitanimura Kazuyuki Tanimura
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: