Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0, 0.22.0
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Gain some insight into blacklist increments/decrements by enhancing the debug logging

      1. MAPREDUCE-2490-branch-0.20-security.patch
        1 kB
        Jonathan Eagles
      2. MAPREDUCE-2490-branch-0.20-security-v2.patch
        4 kB
        Jonathan Eagles
      3. MAPREDUCE-2490-trunk.patch
        0.8 kB
        Jonathan Eagles
      4. MAPREDUCE-2490-trunk-v2.patch
        2 kB
        Jonathan Eagles

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Why is this targeting 0.20 and not trunk?

          Show
          Todd Lipcon added a comment - Why is this targeting 0.20 and not trunk?
          Hide
          Chris Douglas added a comment -

          Why is this targeting 0.20 and not trunk?

          0.20.203 has been displaying negative counts for graylisted node metrics, but it's not clear why. Not sure if it's happening in trunk.

          Show
          Chris Douglas added a comment - Why is this targeting 0.20 and not trunk? 0.20.203 has been displaying negative counts for graylisted node metrics, but it's not clear why. Not sure if it's happening in trunk.
          Hide
          Jonathan Eagles added a comment -

          No new tests for these debug statements. Java doc and eclipse path issues are know branch 20 issues

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no tests are needed for this patch.
          [exec]
          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
          [exec]

          Show
          Jonathan Eagles added a comment - No new tests for these debug statements. Java doc and eclipse path issues are know branch 20 issues [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec]
          Hide
          Jonathan Eagles added a comment -

          @Todd attached trunk patch for blacklisting debug statements while 0.20 security patch contains blacklist and graylist debug statements.

          Show
          Jonathan Eagles added a comment - @Todd attached trunk patch for blacklisting debug statements while 0.20 security patch contains blacklist and graylist debug statements.
          Hide
          Todd Lipcon added a comment -

          Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?

          Show
          Todd Lipcon added a comment - Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479131/MAPREDUCE-2490-trunk.patch
          against trunk revision 1102515.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479131/MAPREDUCE-2490-trunk.patch against trunk revision 1102515. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/244//console This message is automatically generated.
          Hide
          Jonathan Eagles added a comment -

          Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk?

          I don't know of any on-going work to forward port MAPREDUCE-1966. I will look into the history of gray listing to gain some better context to help answer this question.

          Show
          Jonathan Eagles added a comment - Cool, thanks Jonathan. Is work ongoing to forward-port the new greylist functionality to trunk? I don't know of any on-going work to forward port MAPREDUCE-1966 . I will look into the history of gray listing to gain some better context to help answer this question.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch
          against trunk revision 1125428.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479926/MAPREDUCE-2490-trunk-v2.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/287//console This message is automatically generated.
          Hide
          Jonathan Eagles added a comment -

          This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging).

          Show
          Jonathan Eagles added a comment - This most recent patch version still logs the black/gray counts, but has also added some debug info at the caller to better aid in debug the root cause in the case for debugging black/gray list failures. The gridmix contrib test failure is unrelated to my change. No new unit tests were add due to the nature of this change (only affects logging).
          Hide
          Jonathan Eagles added a comment -

          Filed MAPREDUCE-2526 "Add task tracker graylisting" to aid in the discussion to forward port task tracker graylisting to trunk to address comments above.

          Show
          Jonathan Eagles added a comment - Filed MAPREDUCE-2526 "Add task tracker graylisting" to aid in the discussion to forward port task tracker graylisting to trunk to address comments above.
          Hide
          Chris Douglas added a comment -

          +1

          I committed this. Thanks, Jon!

          Show
          Chris Douglas added a comment - +1 I committed this. Thanks, Jon!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #694 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/694/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #686 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/686/)
          MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
          diagnosis of related issues. Contributed by Jonathan Eagles

          cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125588
          Files :

          • /hadoop/mapreduce/trunk/CHANGES.txt
          • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #686 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/686/ ) MAPREDUCE-2490 . Add logging to graylist and blacklist activity to aid diagnosis of related issues. Contributed by Jonathan Eagles cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125588 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.

            People

            • Assignee:
              Jonathan Eagles
              Reporter:
              Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development