XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      HADOOP-13449 adds support for DynamoDBMetadataStore.

      The code currently supports two options for choosing DynamoDB table names:
      1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
      2. Configure a table name in the fs.s3a.s3guard.ddb.table parameter.

      However, if a user sets fs.s3a.s3guard.ddb.table and accesses multiple buckets, DynamoDBMetadataStore does not properly differentiate between paths belonging to different buckets. For example, it would treat s3a://bucket-a/path1 as the same as s3a://bucket-b/path1.

      Goals for this JIRA:

      • Allow for a "one DynamoDB table per cluster" configuration. If a user accesess multiple buckets with that single table, it should work correctly.
      • Explain which credentials are used for DynamoDB. Currently each S3AFileSystem has its own DynamoDBMetadataStore, which uses the credentials from the S3A fs. We at least need to document this behavior.
      • Document any other limitations etc. in the s3guard.md site doc.

        Attachments

        1. HADOOP-13876-HADOOP-13345.005.patch
          23 kB
          Aaron Fabbri
        2. HADOOP-13876-HADOOP-13345.004.patch
          23 kB
          Aaron Fabbri
        3. HADOOP-13876-HADOOP-13345.003.patch
          22 kB
          Aaron Fabbri
        4. HADOOP-13876-HADOOP-13345.002.patch
          25 kB
          Aaron Fabbri
        5. HADOOP-13876-HADOOP-13345.001.patch
          21 kB
          Aaron Fabbri
        6. HADOOP-13876-HADOOP-13345.000.patch
          5 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                fabbri Aaron Fabbri
                Reporter:
                fabbri Aaron Fabbri
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: