VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:


      HADOOP-13449 adds support for DynamoDBMetadataStore.

      The code currently supports two options for choosing DynamoDB table names:
      1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
      2. Configure a table name in the fs.s3a.s3guard.ddb.table parameter.

      However, if a user sets fs.s3a.s3guard.ddb.table and accesses multiple buckets, DynamoDBMetadataStore does not properly differentiate between paths belonging to different buckets. For example, it would treat s3a://bucket-a/path1 as the same as s3a://bucket-b/path1.

      Goals for this JIRA:

      • Allow for a "one DynamoDB table per cluster" configuration. If a user accesess multiple buckets with that single table, it should work correctly.
      • Explain which credentials are used for DynamoDB. Currently each S3AFileSystem has its own DynamoDBMetadataStore, which uses the credentials from the S3A fs. We at least need to document this behavior.
      • Document any other limitations etc. in the site doc.


        1. HADOOP-13876-HADOOP-13345.000.patch
          5 kB
          Mingliang Liu
        2. HADOOP-13876-HADOOP-13345.001.patch
          21 kB
          Aaron Fabbri
        3. HADOOP-13876-HADOOP-13345.002.patch
          25 kB
          Aaron Fabbri
        4. HADOOP-13876-HADOOP-13345.003.patch
          22 kB
          Aaron Fabbri
        5. HADOOP-13876-HADOOP-13345.004.patch
          23 kB
          Aaron Fabbri
        6. HADOOP-13876-HADOOP-13345.005.patch
          23 kB
          Aaron Fabbri

        Issue Links



            • Assignee:
              fabbri Aaron Fabbri
              fabbri Aaron Fabbri


              • Created:

                Issue deployment