[HADOOP-3702] add support for chaining Maps in a single Map and after a Reduce [M*/RM*]

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.0

Environment

all

• Hadoop Flags: Reviewed

Release Note

Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.

• Assignee: Alejandro Abdelnur
• Reporter: Alejandro Abdelnur
• Created: 07/Jul/08 07:41
• Updated: 08/Jul/09 16:52
• Resolved: 09/Sep/08 13:19

Description

On the same input, we usually need to run multiple Maps one after the other without no Reduce. We also have to run multiple Maps after the Reduce.

If all pre-Reduce Maps are chained together and run as a single Map a significant amount of Disk I/O will be avoided.

Similarly all post-Reduce Maps can be chained together and run in the Reduce phase after the Reduce.

Issue Links

• incorporates: HADOOP-3927 Be able to create a Configuration out an stream (with config XML content)