[HADOOP-13904] DynamoDBMetadataStore to handle DDB throttling failures through retry policy

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Assignee: Aaron Fabbri
• Reporter: Steve Loughran
• Created: 14/Dec/16 10:57
• Updated: 20/Feb/17 23:59
• Resolved: 20/Feb/17 23:59

Description

When you overload DDB, you get error messages warning of throttling, as documented by AWS

Reduce load on DDB by doing a table lookup before the create, then, in table create/delete operations and in get/put actions, recognise the error codes and retry using an appropriate retry policy (exponential backoff + ultimate failure)

Issue Links

• is cloned by: HADOOP-14087 S3A typo in pom.xml test exclusions
• is related to: HADOOP-13908 S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
• relates to: HADOOP-13899 tune dynamodb client & tests