[HADOOP-15961] S3A committers: make sure there's regular progress() calls

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Assignee: lqjacklee
• Reporter: Steve Loughran
• Created: 29/Nov/18 17:23
• Updated: 17/Feb/20 22:22
• Resolved: 17/Feb/20 22:09

Description

MAPREDUCE-7164 highlights how inside job/task commit more context.progress() callbacks are needed, just for HDFS.

the S3A committers should be reviewed similarly.

At a glance:

StagingCommitter.commitTaskInternal() is at risk if a task write upload enough data to the localfs that the upload takes longer than the timeout.

it should call progress it every single file commits, or better: modify uploadFileToPendingCommit to take a Progressable for progress callbacks after every part upload.

Issue Links

• relates to: MAPREDUCE-7164 FileOutputCommitter does not report progress while merging paths.
• links to: GitHub Pull Request #1848