[HADOOP-4565] MultiFileInputSplit can use data locality information to create splits

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.20.0
• Hadoop Flags: Reviewed
• Release Note: Improved MultiFileInputFormat so that multiple blocks from the same node or same rack can be combined into a single split.
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 31/Oct/08 22:09
• Updated: 17/Jul/14 19:31
• Resolved: 09/Feb/09 19:47

Description

The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.

Issue Links

• blocks: HIVE-74 Hive can use CombineFileInputFormat for when the input are many small files
• duplicates: HADOOP-2560 Processing multiple input splits per mapper task
• is blocked by: HADOOP-3293 When an input split spans cross block boundary, the split location should be the host having most of bytes on it.
• is blocked by: HADOOP-4567 GetFileBlockLocations should return the NetworkTopology information of the machines that hosts those blocks
• is depended upon by: MAPREDUCE-214 Concrete implementation of MultiFileInputFormat