[HADOOP-1986] Add support for a general serialization mechanism for Map Reduce

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Programs that implement the raw Mapper or Reducer interfaces will need modification to compile with this release. For example,

class MyMapper implements Mapper {
  public void map(WritableComparable key, Writable val,
    OutputCollector out, Reporter reporter) throws IOException {
    // ...
  }
  // ...
}

will need to be changed to refer to the parameterized type. For example:

class MyMapper implements Mapper<WritableComparable, Writable, WritableComparable, Writable> {
  public void map(WritableComparable key, Writable val,
    OutputCollector<WritableComparable, Writable> out, Reporter reporter) throws IOException {
    // ...
  }
  // ...
}

Similarly implementations of the following raw interfaces will need modification: InputFormat, OutputCollector, OutputFormat, Partitioner, RecordReader, RecordWriter

• Assignee: Tom White
• Reporter: Tom White
• Created: 02/Oct/07 21:57
• Updated: 08/Jul/09 17:52
• Resolved: 28/Feb/08 17:47

Description

Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs. While it's possible to write Writable wrappers for other serialization frameworks (such as Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types directly, without explicit wrapping and unwrapping.