[MAPREDUCE-7366] FileOutputCommitter Enable Concurrent Writes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.1
• Component/s: mrv2
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: ismail
• Created: 23/Oct/21 19:41
• Updated: 27/Oct/21 09:18
• Resolved: 26/Oct/21 16:07

Description

is it possible to make `PENDING_DIR_NAME` configurable?
That will enable concurrent writes to same location. current if two spark processes write same destination one of them is failing.

current

 public static final String PENDING_DIR_NAME = "_temporary";

new:

PENDING_DIR_NAME = conf.get("mapreduce.fileoutputcommitter.pending.dir", "_temporary");

here is custom commiter doing it: https://gist.github.com/ismailsimsek/33c55d8e1fcfc79160483c38a978edbd

Issue Links

• duplicates: MAPREDUCE-7331 Make temporary directory used by FileOutputCommitter configurable
• is superceded by: MAPREDUCE-7341 Add a task-manifest output committer for Azure and GCS
• links to: GitHub Pull Request #3582