[HADOOP-15563] S3Guard to support creating on-demand DDB tables - ASF JIRA

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.1.0
Fix Version/s: 3.3.0
Component/s: fs/s3
Labels:
None

Target Version/s:

3.3.0, 3.2.1
Release Note:

Hide
S3Guard now defaults to creating DynamoDB tables as "On-Demand", rather than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata entries, while delivering significantly faster performance during query planning and other bursts of IO. Consult the S3Guard documentation for further details.

Show
S3Guard now defaults to creating DynamoDB tables as "On-Demand", rather than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata entries, while delivering significantly faster performance during query planning and other bursts of IO. Consult the S3Guard documentation for further details.

Description

To keep costs down on DDB, autoscaling is a key feature: you set the max values and when idle, you don't get billed, at the cost of delayed scale time and risk of not getting the max value when AWS is busy

It can be done from the AWS web UI, but not in the s3guard init and set-capacity calls

It can be done through the API

Usual issues then: wiring up, CLI params, testing. It'll be hard to test.

Fully support On-demand DDB tables in S3Guard

create (0, 0) will create an on-demand table.
set capacity (0, 0) will create an on-demand table.
once a table is on demand, any set capacity command other than to (0, 0) will then fail.
when loading table, note if it is on-demand or not
if on demand, prune() doesn't bother to throttle requests any more by sleeping.