Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-112

Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.20.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Updates of counters for reduce input and output records were added in the new API so they are available for jobs using the new API.

      Description

      After running the examples/wordcount (which uses the new API), the reduce input and output record counters always show 0. This is because these counters are not getting updated in the new API

      1. mapred-112-branch20.patch
        9 kB
        Jothi Padmanabhan
      2. mapred-112-10Sep.patch
        9 kB
        Jothi Padmanabhan

        Activity

        Hide
        Arun C Murthy added a comment -

        This is a blocker for 0.20.1.

        Show
        Arun C Murthy added a comment - This is a blocker for 0.20.1.
        Hide
        Jothi Padmanabhan added a comment -

        Straight Forward Patch

        Show
        Jothi Padmanabhan added a comment - Straight Forward Patch
        Hide
        Jothi Padmanabhan added a comment -

        run-test-mapred and contrib tests passed.
        test patch returned +1 for all except Javac warnings where it complained the patch threw 1000 and odd more warnings than trunk – evidently javac checking of test patch is broken.

        Show
        Jothi Padmanabhan added a comment - run-test-mapred and contrib tests passed. test patch returned +1 for all except Javac warnings where it complained the patch threw 1000 and odd more warnings than trunk – evidently javac checking of test patch is broken.
        Hide
        Jothi Padmanabhan added a comment -

        Resubmitting to see if Hudson picks this up

        Show
        Jothi Padmanabhan added a comment - Resubmitting to see if Hudson picks this up
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419186/mapred-112-10Sep.patch
        against trunk revision 813944.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419186/mapred-112-10Sep.patch against trunk revision 813944. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/67/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        Patch for the 20 branch

        Show
        Jothi Padmanabhan added a comment - Patch for the 20 branch
        Hide
        Chris Douglas added a comment -

        Result of test-patch on branch-0.20:

             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        

        +1 I committed this. Thanks, Jothi!

        Show
        Chris Douglas added a comment - Result of test-patch on branch-0.20: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 I committed this. Thanks, Jothi!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/37/)
        . Add counters for reduce input, output records to the new API.
        Contributed by Jothi Padmanabhan

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #37 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/37/ ) . Add counters for reduce input, output records to the new API. Contributed by Jothi Padmanabhan
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #83 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/83/)
        . Add counters for reduce input, output records to the new API.
        Contributed by Jothi Padmanabhan

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #83 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/83/ ) . Add counters for reduce input, output records to the new API. Contributed by Jothi Padmanabhan

          People

          • Assignee:
            Jothi Padmanabhan
            Reporter:
            Jothi Padmanabhan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development