Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.1-beta
    • Fix Version/s: 2.2.0
    • Component/s: mrv1, mrv2
    • Labels:
      None
    • Target Version/s:

      Description

      mapred.lib.TotalOrderPartitioner in branch-1 has these two methods:

      public static String getPartitionFile(JobConf job)
      public static void setPartitionFile(JobConf job, Path p)
      

      In branch-2, mapred.lib.TotalOrderPartitioner is now a subclass of mapred.lib.TotalOrderPartitioner, from which it inherits the similar methods:

      public static String getPartitionFile(Configuration conf)
      public static void setPartitionFile(Configuration conf, Path p)
      

      This means that any code that does either of the following:

      TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
      String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
      

      will not be binary compatible (that is, if compiled against branch-1, it will throw a NoSuchMethodError if run against branch-2).

      1. MAPREDUCE-5529.patch
        1 kB
        Robert Kanter
      2. MAPREDUCE-5529.patch
        2 kB
        Robert Kanter
      3. MAPREDUCE-5529_branch_2.patch
        2 kB
        Arun C Murthy

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        17m 7s 1 Robert Kanter 24/Sep/13 01:18
        Patch Available Patch Available Resolved Resolved
        3d 19h 46m 1 Arun C Murthy 27/Sep/13 21:05
        Resolved Resolved Closed Closed
        640d 11h 9m 1 Vinod Kumar Vavilapalli 30/Jun/15 08:14
        Vinod Kumar Vavilapalli made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Closing old tickets that are already shipped in a release.

        Show
        Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already shipped in a release.
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1562/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1562 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1562/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1536 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1536/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1536 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1536/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #346 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/346/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #346 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/346/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 2.1.2-beta [ 12325050 ]
        Resolution Fixed [ 1 ]
        Hide
        Arun C Murthy added a comment -

        +1. I just committed this, thanks Robert Kanter!

        Thanks for the reviews Zhijie Shen!

        Show
        Arun C Murthy added a comment - +1. I just committed this, thanks Robert Kanter ! Thanks for the reviews Zhijie Shen !
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12605536/MAPREDUCE-5529_branch_2.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4056//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12605536/MAPREDUCE-5529_branch_2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4056//console This message is automatically generated.
        Arun C Murthy made changes -
        Attachment MAPREDUCE-5529_branch_2.patch [ 12605536 ]
        Hide
        Arun C Murthy added a comment -

        Sigh, I needed to modify the branch-2 patch since MAPREDUCE-4594 is only on trunk

        Harsh J: in future, please don't commit simple enhancements such as MAPREDU-4594 only to trunk. Thanks.

        Show
        Arun C Murthy added a comment - Sigh, I needed to modify the branch-2 patch since MAPREDUCE-4594 is only on trunk Harsh J : in future, please don't commit simple enhancements such as MAPREDU-4594 only to trunk. Thanks.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4486 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4486/)
        MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4486 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4486/ ) MAPREDUCE-5529 . Fix compat with hadoop-1 in mapred.TotalOrderPartitioner by re-introducing (get,set)PartitionFile which takes in JobConf. Contributed by Robert Kanter. (acmurthy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527048 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
        Hide
        Zhijie Shen added a comment -

        Vinod Kumar Vavilapalli, we've checked this type of binary incompatibility before. Unfortunately, this one has been missed.

        Anyway, since it's four almost four month before when we checked the incompatibility, IMHO, it's good to check it again in case we missed some thing in the last round, and have some more which was broken afterwards.

        Show
        Zhijie Shen added a comment - Vinod Kumar Vavilapalli , we've checked this type of binary incompatibility before. Unfortunately, this one has been missed. Anyway, since it's four almost four month before when we checked the incompatibility, IMHO, it's good to check it again in case we missed some thing in the last round, and have some more which was broken afterwards.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        If this is indeed breaking binary compat, we'll need to revist a bunch of other things at MAPREDUCE-5108. Zhijie, Robert, did/can one of you test this against a real cluster or just jar?

        Show
        Vinod Kumar Vavilapalli added a comment - If this is indeed breaking binary compat, we'll need to revist a bunch of other things at MAPREDUCE-5108 . Zhijie, Robert, did/can one of you test this against a real cluster or just jar?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604924/MAPREDUCE-5529.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604924/MAPREDUCE-5529.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4040//console This message is automatically generated.
        Hide
        Zhijie Shen added a comment -

        +1 for the latest patch

        Zhijie Shen, can you also look at MAPREDUCE-5530?

        Will do that

        Show
        Zhijie Shen added a comment - +1 for the latest patch Zhijie Shen, can you also look at MAPREDUCE-5530 ? Will do that
        Robert Kanter made changes -
        Attachment MAPREDUCE-5529.patch [ 12604924 ]
        Hide
        Robert Kanter added a comment -

        The new patch marks the methods as deprecated and adds javadoc.

        Zhijie Shen, can you also look at MAPREDUCE-5530? It's another compatibility issue (I created 3: MAPREDUCE-5529, MAPREDUCE-5530, and MAPREDUCE-5531) thanks

        Show
        Robert Kanter added a comment - The new patch marks the methods as deprecated and adds javadoc. Zhijie Shen , can you also look at MAPREDUCE-5530 ? It's another compatibility issue (I created 3: MAPREDUCE-5529 , MAPREDUCE-5530 , and MAPREDUCE-5531 ) thanks
        Hide
        Zhijie Shen added a comment -

        Robert Kanter, like MAPREDUCE-5531, would you please deprecate the added methods, and add the javadoc to pointer the users to the new methods?

        Show
        Zhijie Shen added a comment - Robert Kanter , like MAPREDUCE-5531 , would you please deprecate the added methods, and add the javadoc to pointer the users to the new methods?
        Arun C Murthy made changes -
        Target Version/s 2.1.2-beta [ 12325050 ]
        Hide
        Zhijie Shen added a comment -

        Robert Kanter, thanks for finding out the remaining binary incompatibility issue.

        +1. The patch looks good to me.

        Show
        Zhijie Shen added a comment - Robert Kanter , thanks for finding out the remaining binary incompatibility issue. +1. The patch looks good to me.
        Zhijie Shen made changes -
        Parent MAPREDUCE-5108 [ 12639466 ]
        Issue Type Bug [ 1 ] Sub-task [ 7 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12604702/MAPREDUCE-5529.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604702/MAPREDUCE-5529.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4032//console This message is automatically generated.
        Robert Kanter made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Kanter made changes -
        Description {{mapred.lib.TotalPartitioner}} in branch-1 has these two methods:
        {code:java}
        public static String getPartitionFile(JobConf job)
        public static void setPartitionFile(JobConf job, Path p)
        {code}

        In branch-2, {{mapred.lib.TotalPartitioner}} is now a subclass of {{mapreduce.lib.TotalPartitioner}}, from which it inherits the similar methods:
        {code:java}
        public static String getPartitionFile(Configuration conf)
        public static void setPartitionFile(Configuration conf, Path p)
        {code}

        This means that any code that does either of the following:
        {code:java}
        TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
        String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
        {code}
        will not be binary compatible (that is, if compiled against branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2).
        {{mapred.lib.TotalOrderPartitioner}} in branch-1 has these two methods:
        {code:java}
        public static String getPartitionFile(JobConf job)
        public static void setPartitionFile(JobConf job, Path p)
        {code}

        In branch-2, {{mapred.lib.TotalOrderPartitioner}} is now a subclass of {{mapred.lib.TotalOrderPartitioner}}, from which it inherits the similar methods:
        {code:java}
        public static String getPartitionFile(Configuration conf)
        public static void setPartitionFile(Configuration conf, Path p)
        {code}

        This means that any code that does either of the following:
        {code:java}
        TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path("/"));
        String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
        {code}
        will not be binary compatible (that is, if compiled against branch-1, it will throw a {{NoSuchMethodError}} if run against branch-2).
        Robert Kanter made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-5529.patch [ 12604702 ]
        Hide
        Robert Kanter added a comment -

        The patch adds the two missing methods to mapred.lib.TotalOrderPartitioner; they simply call the new methods in mapreduce.lib.TotalOrderPartitioner because they are source compatible.

        Unit tests aren't really possible for this.

        Show
        Robert Kanter added a comment - The patch adds the two missing methods to mapred.lib.TotalOrderPartitioner ; they simply call the new methods in mapreduce.lib.TotalOrderPartitioner because they are source compatible. Unit tests aren't really possible for this.
        Robert Kanter created issue -

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development