Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1147

Map output records counter missing for map-only jobs in new API

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.20.1, 0.21.0
    • 0.20.2
    • None
    • None
    • Reviewed
    • Adds a counter to track the number of records emitted by map writing directly to HDFS i.e map tasks of job with 0 reducers.

    Description

      In the new API, the counter for map output records is not incremented for map-only jobs

      Attachments

        1. mapred-1147-v1.4-y20.patch
          25 kB
          Amar Kamat
        2. mapred-1147-v1.4.patch
          24 kB
          Amar Kamat
        3. mapred-1147-v1.3.patch
          24 kB
          Amar Kamat

        Activity

          amar_kamat Amar Kamat added a comment -

          Attaching a patch that adds map-output-records to the map-only jobs submitted via new api. Test-patch o/p
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 7 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          All tests passed except TestGridmixSubmission. Debugging further.

          amar_kamat Amar Kamat added a comment - Attaching a patch that adds map-output-records to the map-only jobs submitted via new api. Test-patch o/p [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 7 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. All tests passed except TestGridmixSubmission. Debugging further.
          amar_kamat Amar Kamat added a comment -

          TestGridmixSubmission fails intermittently.

          amar_kamat Amar Kamat added a comment - TestGridmixSubmission fails intermittently.
          amar_kamat Amar Kamat added a comment -

          TestGridMixSubmission failure is a known issue, see MAPREDUCE-1124.

          amar_kamat Amar Kamat added a comment - TestGridMixSubmission failure is a known issue, see MAPREDUCE-1124 .

          Looks good, though NewDirectOutputCollector::reporter can be final and the testcase should use the JUnit4 annotations instead of extending TestCase

          cdouglas Christopher Douglas added a comment - Looks good, though NewDirectOutputCollector::reporter can be final and the testcase should use the JUnit4 annotations instead of extending TestCase
          amar_kamat Amar Kamat added a comment -

          Attaching a new patch that incorporates comments by Chris. Testing in progress.

          amar_kamat Amar Kamat added a comment - Attaching a new patch that incorporates comments by Chris. Testing in progress.
          amar_kamat Amar Kamat added a comment -

          Test patch passed on my box.
          Running through hudson.

          amar_kamat Amar Kamat added a comment - Test patch passed on my box. Running through hudson.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12424579/mapred-1147-v1.4.patch
          against trunk revision 834284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424579/mapred-1147-v1.4.patch against trunk revision 834284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/237/console This message is automatically generated.

          +1 This looks good; would you mind attaching a patch for 0.20?

          cdouglas Christopher Douglas added a comment - +1 This looks good; would you mind attaching a patch for 0.20?
          amar_kamat Amar Kamat added a comment -

          Attaching a patch for branch 20. Result of test-patch
          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 6 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          Testing now.

          amar_kamat Amar Kamat added a comment - Attaching a patch for branch 20. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. Testing now.
          amar_kamat Amar Kamat added a comment -

          For branch 20, all tests except the ones mentioned below have passed.

          1. hdfs.TestDatanodeBlockScanner FAILED (timeout)
          2. hdfs.TestDistributedFileSystem FAILED
          3. hdfs.server.namenode.TestFsck FAILED (timeout)
          4. mapred.TestReduceFetch FAILED

          None of these seems related to this issue.

          amar_kamat Amar Kamat added a comment - For branch 20, all tests except the ones mentioned below have passed. hdfs.TestDatanodeBlockScanner FAILED (timeout) hdfs.TestDistributedFileSystem FAILED hdfs.server.namenode.TestFsck FAILED (timeout) mapred.TestReduceFetch FAILED None of these seems related to this issue.
          amar_kamat Amar Kamat added a comment -

          Also for branch 20, all contrib tests passed except TestHdfsProxy and TestStreamingExitStatus.

          amar_kamat Amar Kamat added a comment - Also for branch 20, all contrib tests passed except TestHdfsProxy and TestStreamingExitStatus.

          I committed this. Thanks, Amar!

          cdouglas Christopher Douglas added a comment - I committed this. Thanks, Amar!
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #120 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/120/)
          . Add map output counters to new API. Contributed by Amar Kamat

          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #120 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/120/ ) . Add map output counters to new API. Contributed by Amar Kamat
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #147 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/147/)

          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #147 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/147/ )

          People

            amar_kamat Amar Kamat
            cdouglas Christopher Douglas
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: