Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-15563

S3Guard to support creating on-demand DDB tables

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.3.0
    • fs/s3
    • None
    • Hide
      S3Guard now defaults to creating DynamoDB tables as "On-Demand", rather than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata entries, while delivering significantly faster performance during query planning and other bursts of IO. Consult the S3Guard documentation for further details.
      Show
      S3Guard now defaults to creating DynamoDB tables as "On-Demand", rather than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata entries, while delivering significantly faster performance during query planning and other bursts of IO. Consult the S3Guard documentation for further details.

    Description

      To keep costs down on DDB, autoscaling is a key feature: you set the max values and when idle, you don't get billed, at the cost of delayed scale time and risk of not getting the max value when AWS is busy

      It can be done from the AWS web UI, but not in the s3guard init and set-capacity calls

      It can be done through the API

      Usual issues then: wiring up, CLI params, testing. It'll be hard to test.

      Fully support On-demand DDB tables in S3Guard

      • create (0, 0) will create an on-demand table.
      • set capacity (0, 0) will create an on-demand table.
      • once a table is on demand, any set capacity command other than to (0, 0) will then fail.
      • when loading table, note if it is on-demand or not
      • if on demand, prune() doesn't bother to throttle requests any more by sleeping.

      Attachments

        Issue Links

          contains

          Sub-task - The sub-task of the issue HADOOP-15888 ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16187 ITestS3GuardToolDynamoDB test failures

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16329 "Wrong Billing mode" test failure in ITestS3GuardToolDynamoDB

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          depends upon

          Sub-task - The sub-task of the issue HADOOP-16117 Update AWS SDK to 1.11.563

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-16118 S3Guard to support on-demand DDB tables

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-15888 ITestDynamoDBMetadataStore can leak (large) DDB tables in test failures/timeout

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #879

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: