[MAPREDUCE-1183] Serializable job components: Mapper, Reducer, InputFormat, OutputFormat et al

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.21.0
• Component/s: client
• Assignee: Owen O'Malley
• Reporter: Arun Murthy
• Created: 05/Nov/09 02:08
• Updated: 26/Jul/12 15:43

Description

Currently the Map-Reduce framework uses Configuration to pass information about the various aspects of a job such as Mapper, Reducer, InputFormat, OutputFormat, OutputCommitter etc. and application developers use org.apache.hadoop.mapreduce.Job.set*Class apis to set them at job-submission time:

Job.setMapperClass(IdentityMapper.class);
Job.setReducerClass(IdentityReducer.class);
Job.setInputFormatClass(TextInputFormat.class);
Job.setOutputFormatClass(TextOutputFormat.class);
...

The proposal is that we move to a model where end-users interact with org.apache.hadoop.mapreduce.Job via actual objects which are then serialized by the framework:

Job.setMapper(new IdentityMapper());
Job.setReducer(new IdentityReducer());
Job.setInputFormat(new TextInputFormat("in"));
Job.setOutputFormat(new TextOutputFormat("out"));
...

Issue Links

• relates to: MAPREDUCE-1462 Enable context-specific and stateful serializers in MapReduce
• relates to: MAPREDUCE-1126 shuffle should use serialization to get comparator