[MAPREDUCE-7470] multi-thread mapreduce v1 FileOutputcommitter

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: mrv2
• Labels: mapreduce pull-request-available
• Assignee: Unassigned
• Reporter: TianyiMa
• Created: 19/Jan/24 14:40
• Updated: 26/Apr/24 03:18
• Resolved: 20/Mar/24 13:32

Description

In cloud environment, such as aws, aliyun etc., the internet delay is non-trival when we commit thounds of files.

In our situation, the ping delay is about 0.03ms in IDC, but when move to Coud, the ping delay is about 3ms, which is roughly 100x slower. We found that, committing tens thounds of files will cost a few tens of minutes. The more files there are, the logger it takes.

So we propose a new committer algorithm, which is a variant of committer algorithm version 1, called 3. In this new algorithm 3, in order to decrease the committer time, we use a thread pool to commit job's final output.

Our test result in Cloud production shows that, the new algorithm 3 has decrease the committer time by serveral tens of times.

Issue Links

• duplicates: MAPREDUCE-7465 performance problem in FileOutputCommitter for big list processed by single thread
• links to: GitHub Pull Request #6469