[HADOOP-17318] S3A committer to support concurrent jobs with same app attempt ID & dest dir

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.3.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/Oct/20 14:10
• Updated: 26/Nov/20 17:29
• Resolved: 26/Nov/20 17:29

Description

Reported failure of magic committer block uploads as pending upload ID is unknown. Likely cause: it's been aborted by another job

1. Make it possible to turn off cleanup of pending uploads in magic committer
2. log more about uploads being deleted in committers
3. and upload ID in the S3aBlockOutputStream errors

There are other concurrency issues when you look close, see SPARK-33230

• magic committer uses app attempt ID as path under __magic; if there are duplicate then they will conflict
• staging committer local temp dir uses app attempt id

Fix will be to have a job UUID which for spark will be picked up from the SPARK-33230 changes, (option to self-generate in job setup for hadoop 3.3.1+ older spark builds); fall back to app-attempt unless that fallback has been disabled

MR: configure to use app attempt ID
Spark: configure to fail job setup if app attempt ID is the source of a job uuid

Issue Links

• causes: HADOOP-17385 ITestS3ADeleteCost.testDirMarkersFileCreation failure
• contains: HADOOP-17066 S3A staging committer committing duplicate files
• is related to: SPARK-31911 Using S3A staging committer, pending uploads are committed more than once and listed incorrectly in _SUCCESS data
• is related to: SPARK-33230 FileOutputWriter jobs have duplicate JobIDs if launched in same second
• is related to: SPARK-33402 Jobs launched in same second have duplicate MapReduce JobIDs
• links to: GitHub Pull Request #2399