Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1147

Map output records counter missing for map-only jobs in new API

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.21.0
    • Fix Version/s: 0.20.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Adds a counter to track the number of records emitted by map writing directly to HDFS i.e map tasks of job with 0 reducers.

      Description

      In the new API, the counter for map output records is not incremented for map-only jobs

      1. mapred-1147-v1.3.patch
        24 kB
        Amar Kamat
      2. mapred-1147-v1.4.patch
        24 kB
        Amar Kamat
      3. mapred-1147-v1.4-y20.patch
        25 kB
        Amar Kamat

        Activity

        Hide
        Amar Kamat added a comment -

        Attaching a patch that adds map-output-records to the map-only jobs submitted via new api. Test-patch o/p
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 7 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        All tests passed except TestGridmixSubmission. Debugging further.

        Show
        Amar Kamat added a comment - Attaching a patch that adds map-output-records to the map-only jobs submitted via new api. Test-patch o/p [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 7 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. All tests passed except TestGridmixSubmission. Debugging further.
        Hide
        Amar Kamat added a comment -

        TestGridmixSubmission fails intermittently.

        Show
        Amar Kamat added a comment - TestGridmixSubmission fails intermittently.
        Hide
        Amar Kamat added a comment -

        TestGridMixSubmission failure is a known issue, see MAPREDUCE-1124.

        Show
        Amar Kamat added a comment - TestGridMixSubmission failure is a known issue, see MAPREDUCE-1124 .
        Hide
        Chris Douglas added a comment -

        Looks good, though NewDirectOutputCollector::reporter can be final and the testcase should use the JUnit4 annotations instead of extending TestCase

        Show
        Chris Douglas added a comment - Looks good, though NewDirectOutputCollector::reporter can be final and the testcase should use the JUnit4 annotations instead of extending TestCase
        Hide
        Amar Kamat added a comment -

        Attaching a new patch that incorporates comments by Chris. Testing in progress.

        Show
        Amar Kamat added a comment - Attaching a new patch that incorporates comments by Chris. Testing in progress.
        Hide
        Amar Kamat added a comment -

        Test patch passed on my box.
        Running through hudson.

        Show
        Amar Kamat added a comment - Test patch passed on my box. Running through hudson.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12424579/mapred-1147-v1.4.patch
        against trunk revision 834284.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424579/mapred-1147-v1.4.patch against trunk revision 834284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        +1 This looks good; would you mind attaching a patch for 0.20?

        Show
        Chris Douglas added a comment - +1 This looks good; would you mind attaching a patch for 0.20?
        Hide
        Amar Kamat added a comment -

        Attaching a patch for branch 20. Result of test-patch
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 6 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        Testing now.

        Show
        Amar Kamat added a comment - Attaching a patch for branch 20. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. Testing now.
        Hide
        Amar Kamat added a comment -

        For branch 20, all tests except the ones mentioned below have passed.

        1. hdfs.TestDatanodeBlockScanner FAILED (timeout)
        2. hdfs.TestDistributedFileSystem FAILED
        3. hdfs.server.namenode.TestFsck FAILED (timeout)
        4. mapred.TestReduceFetch FAILED

        None of these seems related to this issue.

        Show
        Amar Kamat added a comment - For branch 20, all tests except the ones mentioned below have passed. hdfs.TestDatanodeBlockScanner FAILED (timeout) hdfs.TestDistributedFileSystem FAILED hdfs.server.namenode.TestFsck FAILED (timeout) mapred.TestReduceFetch FAILED None of these seems related to this issue.
        Hide
        Amar Kamat added a comment -

        Also for branch 20, all contrib tests passed except TestHdfsProxy and TestStreamingExitStatus.

        Show
        Amar Kamat added a comment - Also for branch 20, all contrib tests passed except TestHdfsProxy and TestStreamingExitStatus.
        Hide
        Chris Douglas added a comment -

        I committed this. Thanks, Amar!

        Show
        Chris Douglas added a comment - I committed this. Thanks, Amar!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #120 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/120/)
        . Add map output counters to new API. Contributed by Amar Kamat

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #120 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/120/ ) . Add map output counters to new API. Contributed by Amar Kamat
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #147 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/147/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #147 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/147/ )

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Chris Douglas
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development