Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Partitioner does not work with the new API. MapTask.java looks for "mapred.partitioner.class" whereas the new API sets it to mapreduce.partitioner.class

      1. h5750.patch
        14 kB
        Owen O'Malley
      2. h5750.patch
        14 kB
        Owen O'Malley
      3. h5750.patch
        15 kB
        Chris Douglas
      4. h5750.patch
        15 kB
        Chris Douglas
      5. h5750.patch
        15 kB
        Chris Douglas
      6. h5750.patch
        16 kB
        Chris Douglas

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        38d 1h 48m 3 Chris Douglas 14/Jul/09 02:17
        Open Open Patch Available Patch Available
        39d 10h 35m 4 Chris Douglas 14/Jul/09 06:17
        Patch Available Patch Available Resolved Resolved
        17h 24m 1 Chris Douglas 14/Jul/09 23:42
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/23/)
        . Fix partitioner to work with new API. Contributed by Owen O'Malley

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/23/ ) . Fix partitioner to work with new API. Contributed by Owen O'Malley
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Owen!

        Show
        Chris Douglas added a comment - I committed this. Thanks, Owen!
        Chris Douglas made changes -
        Attachment h5750.patch [ 12413486 ]
        Hide
        Chris Douglas added a comment -

        Added some documentation. No code changes.

        Show
        Chris Douglas added a comment - Added some documentation. No code changes.
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Chris Douglas added a comment -

        All unit tests passed, save TestStreamingExitStatus, which is known-bad. MAPREDUCE-587

        Show
        Chris Douglas added a comment - All unit tests passed, save TestStreamingExitStatus, which is known-bad. MAPREDUCE-587
        Chris Douglas made changes -
        Attachment h5750.patch [ 12413371 ]
        Hide
        Chris Douglas added a comment -

        This uses a trivial Partitioner for map-only jobs. Verified no javac warnings

             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Chris Douglas added a comment - This uses a trivial Partitioner for map-only jobs. Verified no javac warnings [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Chris Douglas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Chris Douglas added a comment -

        Looking more closely at the patch, it shouldn't call the partitioner for map-only jobs.

        Show
        Chris Douglas added a comment - Looking more closely at the patch, it shouldn't call the partitioner for map-only jobs.
        Chris Douglas made changes -
        Attachment h5750.patch [ 12413366 ]
        Hide
        Chris Douglas added a comment -

        Last patch introduced javac warnings not caught by test-patch; this resolves them.

        Show
        Chris Douglas added a comment - Last patch introduced javac warnings not caught by test-patch; this resolves them.
        Hide
        Chris Douglas added a comment -
             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Chris Douglas added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Chris Douglas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Chris Douglas made changes -
        Attachment h5750.patch [ 12413361 ]
        Hide
        Chris Douglas added a comment -

        Merged with trunk

        Show
        Chris Douglas added a comment - Merged with trunk
        Owen O'Malley made changes -
        Component/s task [ 12312920 ]
        Tom White made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Tom White added a comment -

        Patch needs regenerating following the project split.

        Show
        Tom White added a comment - Patch needs regenerating following the project split.
        Tom White made changes -
        Affects Version/s 0.20.0 [ 12313438 ]
        Fix Version/s 0.20.1 [ 12314047 ]
        Fix Version/s 0.20.1 [ 12313866 ]
        Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
        Key HADOOP-5750 MAPREDUCE-565
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12408525/h5750.patch
        against trunk revision 777594.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/379/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12408525/h5750.patch against trunk revision 777594. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/379/console This message is automatically generated.
        Owen O'Malley made changes -
        Attachment h5750.patch [ 12408524 ]
        Owen O'Malley made changes -
        Attachment h5750.patch [ 12408525 ]
        Hide
        Owen O'Malley added a comment -

        Forgot the --no-prefix. sigh

        Show
        Owen O'Malley added a comment - Forgot the --no-prefix. sigh
        Owen O'Malley made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Owen O'Malley made changes -
        Attachment h5750.patch [ 12408524 ]
        Hide
        Owen O'Malley added a comment -

        A simple git rebase fixed the patch. Git rocks. smile

        Show
        Owen O'Malley added a comment - A simple git rebase fixed the patch. Git rocks. smile
        Sharad Agarwal made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Sharad Agarwal added a comment -

        Please update the patch to trunk.

        Show
        Sharad Agarwal added a comment - Please update the patch to trunk.
        Hide
        Sharad Agarwal added a comment -

        +1 Patch looks good.

        Show
        Sharad Agarwal added a comment - +1 Patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407400/h5750.patch
        against trunk revision 772960.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/308/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407400/h5750.patch against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/308/console This message is automatically generated.
        Owen O'Malley made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Owen O'Malley made changes -
        Attachment h5750.patch [ 12407400 ]
        Hide
        Owen O'Malley added a comment -

        This patch pulls out the partitioner into the MapTask and splits on whether it is a new or old style Mapper. It also moves the test case over to a distributed test case so that it runs with 2 reduces and can detect this problem.

        We really need to fix local test runner to handle multiple reduces.

        Show
        Owen O'Malley added a comment - This patch pulls out the partitioner into the MapTask and splits on whether it is a new or old style Mapper. It also moves the test case over to a distributed test case so that it runs with 2 reduces and can detect this problem. We really need to fix local test runner to handle multiple reduces.
        Owen O'Malley made changes -
        Fix Version/s 0.20.1 [ 12313866 ]
        Priority Major [ 3 ] Blocker [ 1 ]
        Owen O'Malley made changes -
        Field Original Value New Value
        Assignee Owen O'Malley [ owen.omalley ]
        Jothi Padmanabhan created issue -

          People

          • Assignee:
            Owen O'Malley
            Reporter:
            Jothi Padmanabhan
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development