Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5108

Changes needed for Binary Compatibility for MR applications via YARN

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly.

      We've done various pieces of work over time, let's track progress and document things clearly. Zhijie Shen has done a bunch of testing and results look very promising so far.

      The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work.

      Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible.

      Also, it's clear that we will not be able to support org.apache.hadoop.mapreduce api in a binary compatible manner due to the interface changes we made in hadoop-0.21 (sigh), and hence, users using the new apis will have to re-compile (i.e. source compatible only).

      Net, given that vast majority of users use the org.apache.hadoop.mapred api, it's a very reasonable way to ease migration to hadoop-2.

        Attachments

        1. Binary Backward Compatibility.pdf
          224 kB
          Zhijie Shen
        2. MR_API_DIFF_v2.tar.gz
          3.05 MB
          Zhijie Shen
        3. mr1_mr2_api_diff.tar.gz
          3.03 MB
          Zhijie Shen
        There are no Sub-Tasks for this issue.

          Activity

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              acmurthy Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: