Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0, 0.22.0
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Gain some insight into blacklist increments/decrements by enhancing the debug logging

      1. MAPREDUCE-2490-trunk-v2.patch
        2 kB
        Jonathan Eagles
      2. MAPREDUCE-2490-branch-0.20-security-v2.patch
        4 kB
        Jonathan Eagles
      3. MAPREDUCE-2490-trunk.patch
        0.8 kB
        Jonathan Eagles
      4. MAPREDUCE-2490-branch-0.20-security.patch
        1 kB
        Jonathan Eagles

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          7d 1h 1m 1 Jonathan Eagles 20/May/11 17:55
          Open Open Patch Available Patch Available
          19h 19m 2 Jonathan Eagles 20/May/11 17:56
          Patch Available Patch Available Resolved Resolved
          5h 58m 1 Chris Douglas 20/May/11 23:55
          Resolved Resolved Closed Closed
          104d 23h 18m 1 Owen O'Malley 02/Sep/11 23:13
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
          Owen O'Malley made changes -
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Owen O'Malley made changes -
          Fix Version/s 0.20.204.0 [ 12316318 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #686 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/686/)
          MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
          diagnosis of related issues. Contributed by Jonathan Eagles

          cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125588
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #686 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/686/ ) MAPREDUCE-2490 . Add logging to graylist and blacklist activity to aid diagnosis of related issues. Contributed by Jonathan Eagles cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125588 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/ )
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Resolution Fixed [ 1 ]
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Jon!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Jon!
          Hide
          Jonathan Eagles added a comment -

          Filed MAPREDUCE-2526 "Add task tracker graylisting" to aid in the discussion to forward port task tracker graylisting to trunk to address comments above.

          Show
          Jonathan Eagles added a comment - Filed MAPREDUCE-2526 "Add task tracker graylisting" to aid in the discussion to forward port task tracker graylisting to trunk to address comments above.
          Hide
          Jonathan Eagles added a comment -

          This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging).

          Show
          Jonathan Eagles added a comment - This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch
          against trunk revision 1125428.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console This message is automatically generated.
          Jonathan Eagles made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jonathan Eagles made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jonathan Eagles made changes -
          Attachment MAPREDUCE-2490-trunk-v2.patch [ 12479926 ]
          Jonathan Eagles made changes -
          Jonathan Eagles made changes -
          Attachment MAPREDUCE-2490-branch-0.20-security-v2.patch [ 12479924 ]
          Jonathan Eagles made changes -
          Attachment MAPREDUCE-2490-branch-0.20-security-v2.patch [ 12479924 ]
          Hide
          Jonathan Eagles added a comment -

          Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?

          I don't know of any on-going work to forward port MAPREDUCE-1966. I will look into the history of gray listing to gain some better context to help answer this question.

          Show
          Jonathan Eagles added a comment - Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk? I don't know of any on-going work to forward port MAPREDUCE-1966 . I will look into the history of gray listing to gain some better context to help answer this question.
          Jonathan Eagles made changes -
          Link This issue is related to MAPREDUCE-1966 [ MAPREDUCE-1966 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479131/MAPREDUCE-2490-trunk.patch
          against trunk revision 1102515.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479131/MAPREDUCE-2490-trunk.patch against trunk revision 1102515. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?

          Show
          Todd Lipcon added a comment - Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?
          Hide
          Jonathan Eagles added a comment -

          @Todd attached trunk patch for blacklisting debug statements while 0.20 security patch contains blacklist and graylist debug statements.

          Show
          Jonathan Eagles added a comment - @Todd attached trunk patch for blacklisting debug statements while 0.20 security patch contains blacklist and graylist debug statements.
          Hide
          Jonathan Eagles added a comment -

          No new tests for these debug statements. Java doc and eclipse path issues are know branch 20 issues

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no tests are needed for this patch.
          [exec]
          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
          [exec]

          Show
          Jonathan Eagles added a comment - No new tests for these debug statements. Java doc and eclipse path issues are know branch 20 issues [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec]
          Jonathan Eagles made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Jonathan Eagles made changes -
          Attachment MAPREDUCE-2490-trunk.patch [ 12479131 ]
          Jonathan Eagles made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-2490-branch-0.20-security.patch [ 12479130 ]
          Hide
          Chris Douglas added a comment -

          Why is this targeting 0.20 and not trunk?

          0.20.203 has been displaying negative counts for graylisted node metrics, but it's not clear why. Not sure if it's happening in trunk.

          Show
          Chris Douglas added a comment - Why is this targeting 0.20 and not trunk? 0.20.203 has been displaying negative counts for graylisted node metrics, but it's not clear why. Not sure if it's happening in trunk.
          Hide
          Todd Lipcon added a comment -

          Why is this targeting 0.20 and not trunk?

          Show
          Todd Lipcon added a comment - Why is this targeting 0.20 and not trunk?
          Jonathan Eagles created issue -

            People

            • Assignee:
              Jonathan Eagles
              Reporter:
              Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development