Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta
    • Fix Version/s: 2.2.0
    • Component/s: mrv1, mrv2
    • Labels:
      None
    • Target Version/s:

      Description

      mapred.lib.TotalOrderPartitioner in branch-1 has these two methods:

      public static String getPartitionFile(JobConf job)
      public static void setPartitionFile(JobConf job, Path p)
      

      In branch-2, mapred.lib.TotalOrderPartitioner is now a subclass of mapred.lib.TotalOrderPartitioner, from which it inherits the similar methods:

      public static String getPartitionFile(Configuration conf)
      public static void setPartitionFile(Configuration conf, Path p)
      

      This means that any code that does either of the following:

      TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
      String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
      

      will not be binary compatible (that is, if compiled against branch-1, it will throw a NoSuchMethodError if run against branch-2).

      1. MAPREDUCE-5529_branch_2.patch
        2 kB
        Arun C Murthy
      2. MAPREDUCE-5529.patch
        2 kB
        Robert Kanter
      3. MAPREDUCE-5529.patch
        1 kB
        Robert Kanter

        Activity

        Hide
        Robert Kanter added a comment -

        The patch adds the two missing methods to mapred.lib.TotalOrderPartitioner; they simply call the new methods in mapreduce.lib.TotalOrderPartitioner because they are source compatible.

        Unit tests aren't really possible for this.

        Show
        Robert Kanter added a comment - The patch adds the two missing methods to mapred.lib.TotalOrderPartitioner ; they simply call the new methods in mapreduce.lib.TotalOrderPartitioner because they are source compatible. Unit tests aren't really possible for this.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604702/MAPREDUCE-5529.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604702/MAPREDUCE-5529.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//console This message is automatically generated.
        Hide
        Zhijie Shen added a comment -

        Robert Kanter, thanks for finding out the remaining binary incompatibility issue.

        +1. The patch looks good to me.

        Show
        Zhijie Shen added a comment - Robert Kanter , thanks for finding out the remaining binary incompatibility issue. +1. The patch looks good to me.
        Hide
        Zhijie Shen added a comment -

        Robert Kanter, like MAPREDUCE-5531, would you please deprecate the added methods, and add the javadoc to pointer the users to the new methods?

        Show
        Zhijie Shen added a comment - Robert Kanter , like MAPREDUCE-5531 , would you please deprecate the added methods, and add the javadoc to pointer the users to the new methods?
        Hide
        Robert Kanter added a comment -

        The new patch marks the methods as deprecated and adds javadoc.

        Zhijie Shen, can you also look at MAPREDUCE-5530? It's another compatibility issue (I created 3: MAPREDUCE-5529, MAPREDUCE-5530, and MAPREDUCE-5531) thanks

        Show
        Robert Kanter added a comment - The new patch marks the methods as deprecated and adds javadoc. Zhijie Shen , can you also look at MAPREDUCE-5530 ? It's another compatibility issue (I created 3: MAPREDUCE-5529 , MAPREDUCE-5530 , and MAPREDUCE-5531 ) thanks
        Hide
        Zhijie Shen added a comment -

        +1 for the latest patch

        Zhijie Shen, can you also look at MAPREDUCE-5530?

        Will do that

        Show
        Zhijie Shen added a comment - +1 for the latest patch Zhijie Shen, can you also look at MAPREDUCE-5530 ? Will do that
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604924/MAPREDUCE-5529.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604924/MAPREDUCE-5529.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        If this is indeed breaking binary compat, we'll need to revist a bunch of other things at MAPREDUCE-5108. Zhijie, Robert, did/can one of you test this against a real cluster or just jar?

        Show
        Vinod Kumar Vavilapalli added a comment - If this is indeed breaking binary compat, we'll need to revist a bunch of other things at MAPREDUCE-5108 . Zhijie, Robert, did/can one of you test this against a real cluster or just jar?
        Hide
        Zhijie Shen added a comment -

        Vinod Kumar Vavilapalli, we've checked this type of binary incompatibility before. Unfortunately, this one has been missed.

        Anyway, since it's four almost four month before when we checked the incompatibility, IMHO, it's good to check it again in case we missed some thing in the last round, and have some more which was broken afterwards.

        Show
        Zhijie Shen added a comment - Vinod Kumar Vavilapalli , we've checked this type of binary incompatibility before. Unfortunately, this one has been missed. Anyway, since it's four almost four month before when we checked the incompatibility, IMHO, it's good to check it again in case we missed some thing in the last round, and have some more which was broken afterwards.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4486 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4486/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4486 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4486/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Arun C Murthy added a comment -

        Sigh, I needed to modify the branch-2 patch since MAPREDUCE-4594 is only on trunk

        Harsh J: in future, please don't commit simple enhancements such as MAPREDU-4594 only to trunk. Thanks.

        Show
        Arun C Murthy added a comment - Sigh, I needed to modify the branch-2 patch since MAPREDUCE-4594 is only on trunk Harsh J : in future, please don't commit simple enhancements such as MAPREDU-4594 only to trunk. Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605536/MAPREDUCE-5529_branch_2.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4056//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605536/MAPREDUCE-5529_branch_2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4056//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        +1. I just committed this, thanks Robert Kanter!

        Thanks for the reviews Zhijie Shen!

        Show
        Arun C Murthy added a comment - +1. I just committed this, thanks Robert Kanter ! Thanks for the reviews Zhijie Shen !
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #346 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/346/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #346 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/346/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1536 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1536/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1536 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1536/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1562/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1562/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development