Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-13960

Initialize DynamoDBMetadataStore without associated S3AFileSystem

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HADOOP-13345
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Per the discussion in HADOOP-13650, it's helpful to initialize a DynamoDBMetadataStore object without S3AFileSystem. In the current code, we can achieve this via DynamoDBMetadataStore#initialize(Configuration). However, users still have to provide the associated S3AFileSystem URI in the configuration, by means of either setting the fs.defaultFS in configuration file or -fs s3://bucket command line parameter. Setting the default FileSystem in configuration seems not necessary as the command line is to manipulate metadata store, e.g. command line tools on an existing HDFS cluster.

      This JIRA is to track the effort of initializing a DynamoDBMetadataStore without associating any S3 buckets, so that S3AFileSystem is not needed. Users have to specify in configuration the DynamoDB endpoints (for region) and table name along with credentials, AWS client settings.

      See Lei (Eddy) Xu and Mingliang Liu's comments in HADOOP-13650 for more details.

        Attachments

          Activity

            People

            • Assignee:
              liuml07 Mingliang Liu
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: