Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4771

KeyFieldBasedPartitioner not partitioning properly when configured

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.1-alpha
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.5
    • Component/s: mrv2
    • Labels:
      None

      Description

      Relative to Hadoop 0.20/1.x, KeyFieldBasedPartitioner is not distributing across partitions properly when configured. This is related to the double-configure issue as described in HADOOP-7425. KeyFieldBasedPartitioner is getting configured twice, and that ends up duplicating the keyspecs and causing the keys to be hashed twice.

      KeyFieldBasedPartitioner should not duplicate keyspecs when configured twice.

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          The regression from 1.x occurs because KeyFieldBasedPartitioner implements Configurable and JobConfigurable. In 1.x it only implemented JobConfigurable, so it was only configured once.

          Show
          Jason Lowe added a comment - The regression from 1.x occurs because KeyFieldBasedPartitioner implements Configurable and JobConfigurable. In 1.x it only implemented JobConfigurable, so it was only configured once.
          Hide
          Jason Lowe added a comment -

          Patch to allow KeyFieldBasedPartitioner to be configured multiple times and setup itself based on the last configuration specified.

          Show
          Jason Lowe added a comment - Patch to allow KeyFieldBasedPartitioner to be configured multiple times and setup itself based on the last configuration specified.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12552159/MAPREDUCE-4771.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.TestClusterMRNotification

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2988//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2988//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552159/MAPREDUCE-4771.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestClusterMRNotification +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2988//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2988//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          TestClusterMRNotification has been failing for a while now. The changes look good to me. +1 I'll check it in.

          Show
          Robert Joseph Evans added a comment - TestClusterMRNotification has been failing for a while now. The changes look good to me. +1 I'll check it in.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2956 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2956/)
          MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2956 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2956/ ) MAPREDUCE-4771 . KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks Jason,

          I put this in trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks Jason, I put this in trunk, branch-2, and branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #28 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/28/)
          MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #28 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/28/ ) MAPREDUCE-4771 . KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #427 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/427/)
          svn merge -c 1405975. FIXES: MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405996)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405996
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #427 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/427/ ) svn merge -c 1405975. FIXES: MAPREDUCE-4771 . KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405996) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405996 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1218/)
          MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1218 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1218/ ) MAPREDUCE-4771 . KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1248 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1248/)
          MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1248 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1248/ ) MAPREDUCE-4771 . KeyFieldBasedPartitioner not partitioning properly when configured (jlowe via bobby) (Revision 1405975) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1405975 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/KeyFieldBasedPartitioner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/lib/TestKeyFieldBasedPartitioner.java

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Jason Lowe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development