[MAPREDUCE-5108] Changes needed for Binary Compatibility for MR applications via YARN

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.3-alpha
• Target Version/s: 2.1.0-beta
• Assignee: Zhijie Shen
• Reporter: Arun Murthy
• Created: 27/Mar/13 21:33
• Updated: 16/Aug/13 04:13
• Resolved: 18/Jun/13 16:43

Description

As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly.

We've done various pieces of work over time, let's track progress and document things clearly. zjshen has done a bunch of testing and results look very promising so far.

The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work.

Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible.

Also, it's clear that we will not be able to support org.apache.hadoop.mapreduce api in a binary compatible manner due to the interface changes we made in hadoop-0.21 (sigh), and hence, users using the new apis will have to re-compile (i.e. source compatible only).

Net, given that vast majority of users use the org.apache.hadoop.mapred api, it's a very reasonable way to ease migration to hadoop-2.

Sub-Tasks

• Hadoop-examples-1.x.x.jar cannot run on Yarn: Closed
• Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib: Closed
• Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate: Closed
• Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn: Closed
• Document MR Binary Compatibility vis-a-vis hadoop-1 and hadoop-2: Closed
• JobStatus#getJobPriority changed to JobStatus#getPriority in MR2: Resolved
• Mapred API: TaskCompletionEvent incompatibility issues with MR1: Closed
• Fix JobClient incompatibilities with MR1: Closed
• Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce: Closed
• TEMP_DIR_NAME is removed from of FileOutputCommitter of mapreduce: Closed
• createFileSplit is removed from NLineInputFormat of mapred: Closed
• Constructor of DBInputFormat.DBRecordReader in mapred is changed: Closed
• Functions are changed or removed from Job in jobcontrol: Closed
• Signature changes for getTaskId of TaskReport in mapred: Closed
• mapred.Counters incompatiblity issues with MR1: Closed
• ClusterStatus incompatiblity issues with MR1: Closed
• mapreduce.Job has a bunch of methods that throw InterruptedException so its incompatible with MR1: Closed
• mapreduce.Job is missing getJobClient() so its incompatible with MR1: Resolved
• API Incompatibility - Sampler: Resolved
• MRAdmin is removed from M/R while RMAdmin is added to Yarn: Closed
• Compatibility: Add a deprecated MRAdmin that wraps around RMAdmin: Resolved
• Two functions changed their visibility in JobStatus: Closed
• A number of public static variables are removed from JobConf: Closed
• filecache.DistributedCache incompatiblity issues with MR1: Closed
• Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce: Closed
• Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils: Closed
• Mapreduce API: TokenCache incompatibility issues with MR1: Closed
• Mapreduce API: ClusterMetrics incompatibility issues with MR1: Closed
• Mapreduce API: Counter changes from non-abstract class to interface: Resolved
• Mapreduce API: CounterGroup changes from non-abstract class to interface: Resolved
• Mapred API: Function signature change in JobControl: Closed
• Mapred API: void setTaskID(TaskAttemptID) is missing in TaskCompletionEvent: Closed
• Two function signature changes in filecache.DistributedCache: Closed
• mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have incompatible signature changes: Closed
• Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2: Closed
• Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2: Closed
• Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2: Closed
• Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes: Closed