[HADOOP-15604] Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Gabor Bota
• Created: 13/Jul/18 09:05
• Updated: 20/Jun/19 11:00
• Resolved: 20/Jun/19 11:00

Description

When there are ~50 files being committed; each in their own thread from the commit pool; probably the DDB repo is being overloaded just from one single process doing task commit. We should be backing off more, especially given that failing on a write could potentially leave the store inconsistent with the FS (renames, etc)

It would be nice to have some tests to prove that the I/O thresholds are the reason for unprocessed items in DynamoDB metadata store

Issue Links

• Is contained by: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename
• is depended upon by: HADOOP-14936 S3Guard: remove "experimental" from documentation
• is related to: HADOOP-15349 S3Guard DDB retryBackoff to be more informative on limits exceeded
• relates to: HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures
• relates to: HADOOP-15576 S3A Multipart Uploader to work with S3Guard and encryption
• relates to: HADOOP-16118 S3Guard to support on-demand DDB tables