[HADOOP-16349] DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Component/s: fs/s3
• Assignee: Gabor Bota
• Reporter: Steve Loughran
• Created: 05/Jun/19 12:12
• Updated: 11/Oct/19 10:10
• Resolved: 11/Oct/19 10:10

Description

If you delete the version marker from a S3Guard table, it appears to hang for 5 minutes.

Only if you restart and turn logging to debug do you see that DynamoDBMetadataStore.getVersionMarkerItem() is sleeping and retrying.

1. log at warn
2. add entry to troubleshooting doc on the topic

The cause of the failure can be any of

• table being inited elsewhere: expectation, fast recovery
• it's not a S3Guard table: it won't recover
• it's a S3Guard table without a version marker: it won't recover.

+ consider having a shorter retry lifespan, though if it adds a new config point I'm a bit reluctant. For s3guard bucket-info it would make sense to change the policy to be aggressively short lived

Issue Links

• is related to: HADOOP-16520 Race condition in DDB table init and waiting threads