[HADOOP-3258] FileOutputFormat should have a method to create custom files under the outputdir with a unique name per task to avoid name collision

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate

Environment

all

• Release Note: FileOutputFormat provides a new static method, getPathForCustomFile, that creates a custom filename under the outputdir that is namespace with the task ID and task type (i.e. myfile-r-00001).
• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Created: 15/Apr/08 07:57
• Updated: 08/Jul/09 16:52
• Resolved: 11/Jul/08 03:05

Description

Currently, if a M/R code creates a file, it is the responsibility of the M/R code to avoid file name collisions from different tasks.

Hadoop should provide an API that creates unique file names based on the task type (map or reduce) and the task ID. Similarly to how output files, part-#####, are created.

The proposed patch adds 2 static methods to the FileOutputFormat

{nofomat}
public static String getUniqueName(JobConf conf, String name);
public static Path getPathForCustomFile(JobConf conf, String name);{nofomat}

The first one adds task type and task ID to the given name.

The second gives a PATH to a file in the working outputdir using a file name namespaced by the first method.

Issue Links

• is part of: HADOOP-3149 supporting multiple outputs for M/R jobs