MAPREDUCE-7164 highlights how inside job/task commit more context.progress() callbacks are needed, just for HDFS.
the S3A committers should be reviewed similarly.
At a glance:
StagingCommitter.commitTaskInternal() is at risk if a task write upload enough data to the localfs that the upload takes longer than the timeout.
it should call progress it every single file commits, or better: modify uploadFileToPendingCommit to take a Progressable for progress callbacks after every part upload.
- relates to
MAPREDUCE-7164 FileOutputCommitter does not report progress while merging paths.
- links to