Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2456

Show the reducer taskid and map/reduce tasktrackers for "Failed fetch notification #_ for task attempt..." log messages

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira is to provide more useful log information for debugging the "Too many fetch-failures" error.

      Looking at the JobTracker node, we see messages like this:
      "2010-12-14 00:00:06,911 INFO org.apache.hadoop.mapred.JobInProgress: Failed fetch notification #8 for task
      attempt_201011300729_189729_m_007458_0".

      I would be useful to see which reducer is reporting the error here.

      So, I propose we add the following to these log messages:
      1. reduce task ID
      2. TaskTracker nodenames for both the mapper and the reducer

      1. MAPREDUCE-2456-trunk.patch
        3 kB
        Jeffrey Naisbitt
      2. MAPREDUCE-2456-0.20s.patch
        3 kB
        Jeffrey Naisbitt
      3. MAPREDUCE-2456-trunk.patch
        3 kB
        Jeffrey Naisbitt

        Activity

        Hide
        Jeffrey Naisbitt added a comment -

        Patch for the 0.20-security branch

        Show
        Jeffrey Naisbitt added a comment - Patch for the 0.20-security branch
        Hide
        Jeffrey Naisbitt added a comment -

        No tests were included since this is basically just adding information to log messages.

        Here are the test-patch results from the 0.20-security branch. The javadoc and Eclipse classpath errors are unrelated to this bug since they appear in the 0.20-security branch without my changes.
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no tests are needed for this patch.
        [exec]
        [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec]
        [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec]
        [exec]
        [exec]

        Show
        Jeffrey Naisbitt added a comment - No tests were included since this is basically just adding information to log messages. Here are the test-patch results from the 0.20-security branch. The javadoc and Eclipse classpath errors are unrelated to this bug since they appear in the 0.20-security branch without my changes. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec] [exec] [exec]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12478210/MAPREDUCE-2456-0.20s.patch
        against trunk revision 1099590.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/214//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478210/MAPREDUCE-2456-0.20s.patch against trunk revision 1099590. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/214//console This message is automatically generated.
        Hide
        Jeffrey Naisbitt added a comment -

        I must have attached my patch for the 0.20-security branch too soon (or I did something else wrong) since the automated test-patch run above tried to use the wrong patch file. The above error was because it tried to apply the 0.20-security patch to trunk.

        Show
        Jeffrey Naisbitt added a comment - I must have attached my patch for the 0.20-security branch too soon (or I did something else wrong) since the automated test-patch run above tried to use the wrong patch file. The above error was because it tried to apply the 0.20-security patch to trunk.
        Hide
        Jeffrey Naisbitt added a comment -

        Here is my test-patch output when run on trunk. Again, no tests were included because it's just changing log messages

        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 system test framework. The patch passed system test framework compile.

        Show
        Jeffrey Naisbitt added a comment - Here is my test-patch output when run on trunk. Again, no tests were included because it's just changing log messages [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile.
        Hide
        Jeffrey Naisbitt added a comment -

        Reattaching patch to get it to run through the automated Hudson tests

        Show
        Jeffrey Naisbitt added a comment - Reattaching patch to get it to run through the automated Hudson tests
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12478616/MAPREDUCE-2456-trunk.patch
        against trunk revision 1100409.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.mapred.TestDebugScript

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478616/MAPREDUCE-2456-trunk.patch against trunk revision 1100409. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestDebugScript +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/226//console This message is automatically generated.
        Hide
        Jeffrey Naisbitt added a comment -

        The failing core test is unrelated to this patch, and it passes for me when run manually:

        [junit] Running org.apache.hadoop.mapred.TestDebugScript
        [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 8.085 sec

        Show
        Jeffrey Naisbitt added a comment - The failing core test is unrelated to this patch, and it passes for me when run manually: [junit] Running org.apache.hadoop.mapred.TestDebugScript [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 8.085 sec
        Hide
        Chris Douglas added a comment -

        +1

        I committed this to MR trunk and the 20-security branch. Thanks, Jeff!

        Show
        Chris Douglas added a comment - +1 I committed this to MR trunk and the 20-security branch. Thanks, Jeff!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #661 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/661/)
        MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
        failed fetch notifications in the JobTracker log.
        Contributed by Jeffrey Naisbitt

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #661 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/661/ ) MAPREDUCE-2456 . Log the reduce taskID and associated TaskTrackers with failed fetch notifications in the JobTracker log. Contributed by Jeffrey Naisbitt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #676 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/676/)
        MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
        failed fetch notifications in the JobTracker log.
        Contributed by Jeffrey Naisbitt

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #676 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/676/ ) MAPREDUCE-2456 . Log the reduce taskID and associated TaskTrackers with failed fetch notifications in the JobTracker log. Contributed by Jeffrey Naisbitt
        Hide
        Owen O'Malley added a comment -

        Hadoop 0.20.204.0 was just released.

        Show
        Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.

          People

          • Assignee:
            Jeffrey Naisbitt
            Reporter:
            Jeffrey Naisbitt
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development