Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Partitioner does not work with the new API. MapTask.java looks for "mapred.partitioner.class" whereas the new API sets it to mapreduce.partitioner.class

      1. h5750.patch
        14 kB
        Owen O'Malley
      2. h5750.patch
        14 kB
        Owen O'Malley
      3. h5750.patch
        15 kB
        Chris Douglas
      4. h5750.patch
        15 kB
        Chris Douglas
      5. h5750.patch
        15 kB
        Chris Douglas
      6. h5750.patch
        16 kB
        Chris Douglas

        Activity

        Hide
        Owen O'Malley added a comment -

        This patch pulls out the partitioner into the MapTask and splits on whether it is a new or old style Mapper. It also moves the test case over to a distributed test case so that it runs with 2 reduces and can detect this problem.

        We really need to fix local test runner to handle multiple reduces.

        Show
        Owen O'Malley added a comment - This patch pulls out the partitioner into the MapTask and splits on whether it is a new or old style Mapper. It also moves the test case over to a distributed test case so that it runs with 2 reduces and can detect this problem. We really need to fix local test runner to handle multiple reduces.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407400/h5750.patch
        against trunk revision 772960.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/308/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407400/h5750.patch against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/308/console This message is automatically generated.
        Hide
        Sharad Agarwal added a comment -

        +1 Patch looks good.

        Show
        Sharad Agarwal added a comment - +1 Patch looks good.
        Hide
        Sharad Agarwal added a comment -

        Please update the patch to trunk.

        Show
        Sharad Agarwal added a comment - Please update the patch to trunk.
        Hide
        Owen O'Malley added a comment -

        A simple git rebase fixed the patch. Git rocks. smile

        Show
        Owen O'Malley added a comment - A simple git rebase fixed the patch. Git rocks. smile
        Hide
        Owen O'Malley added a comment -

        Forgot the --no-prefix. sigh

        Show
        Owen O'Malley added a comment - Forgot the --no-prefix. sigh
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12408525/h5750.patch
        against trunk revision 777594.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/379/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408525/h5750.patch against trunk revision 777594. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/379/console This message is automatically generated.
        Hide
        Tom White added a comment -

        Patch needs regenerating following the project split.

        Show
        Tom White added a comment - Patch needs regenerating following the project split.
        Hide
        Chris Douglas added a comment -

        Merged with trunk

        Show
        Chris Douglas added a comment - Merged with trunk
        Hide
        Chris Douglas added a comment -
             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Chris Douglas added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Chris Douglas added a comment -

        Last patch introduced javac warnings not caught by test-patch; this resolves them.

        Show
        Chris Douglas added a comment - Last patch introduced javac warnings not caught by test-patch; this resolves them.
        Hide
        Chris Douglas added a comment -

        Looking more closely at the patch, it shouldn't call the partitioner for map-only jobs.

        Show
        Chris Douglas added a comment - Looking more closely at the patch, it shouldn't call the partitioner for map-only jobs.
        Hide
        Chris Douglas added a comment -

        This uses a trivial Partitioner for map-only jobs. Verified no javac warnings

             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Chris Douglas added a comment - This uses a trivial Partitioner for map-only jobs. Verified no javac warnings [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Chris Douglas added a comment -

        All unit tests passed, save TestStreamingExitStatus, which is known-bad. MAPREDUCE-587

        Show
        Chris Douglas added a comment - All unit tests passed, save TestStreamingExitStatus, which is known-bad. MAPREDUCE-587
        Hide
        Chris Douglas added a comment -

        Added some documentation. No code changes.

        Show
        Chris Douglas added a comment - Added some documentation. No code changes.
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Owen!

        Show
        Chris Douglas added a comment - I committed this. Thanks, Owen!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/23/)
        . Fix partitioner to work with new API. Contributed by Owen O'Malley

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/23/ ) . Fix partitioner to work with new API. Contributed by Owen O'Malley

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Jothi Padmanabhan
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development