[HADOOP-16458] LocatedFileStatusFetcher scans failing intermittently against S3 store

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3

Environment

S3 + S3Guard

• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 24/Jul/19 17:29
• Updated: 14/Jul/21 17:26
• Resolved: 01/Oct/19 17:13

Description

Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is using globStatus to find files.

I'd say "turn s3guard on" except this appears to be the case, and the dataset being read is
over 1h old.

Which means it is harder than I'd like to blame S3 for what would sound like an inconsistency

We're hampered by the number of debug level statements in the globber code being approximately none; there's no debugging to turn on. All we know is that globFiles returns null without any explanation.

Issue Links

• breaks: HIVE-24484 Upgrade Hadoop to 3.3.1 And Tez to 0.10.2
• causes: HADOOP-16626 S3A ITestRestrictedReadAccess fails
• causes: HADOOP-17105 S3AFS globStatus attempts to resolve symlinks
• contains: HADOOP-13373 Add S3A implementation of FSMainOperationsBaseTest
• is depended upon by: HADOOP-16465 Tune S3AFileSystem.listLocatedStatus
• relates to: HADOOP-15748 S3 listing inconsistency can raise NPE in globber
• supercedes: HADOOP-13371 S3A globber to use bulk listObject call over recursive directory scan
• links to: GitHub Pull Request #1160