[MAPREDUCE-6208] There should be an input format for MapFiles which can be configured so that only a fraction of the input data is used for the MR process

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Labels: BB2015-05-TBR inputformat mapfile
• Assignee: Jens Rabe
• Reporter: Jens Rabe
• Created: 27/Dec/14 10:49
• Updated: 06/May/15 03:27

Description

In some cases there are large amounts of data organized in MapFiles, e.g., from previous MapReduce tasks, and only a fraction of the data is to be processed in a MR task. The current approach, as I understand, is to re-organize the data in a suitable partition using folders on HDFS, and only use relevant folders as input paths, and maybe doing some additional filtering in the Map task. However, sometimes the input data cannot be easily partitioned that way. For example, when processing large amounts of measured data where additional data on a time period already in HDFS arrives later.

There should be an input format that accepts folders with MapFiles, and there should be an option to specify the input key range so that only fitting InputSplits are generated.