Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2901

Add errors and warning metrics page to RM, NM web UI

    Details

    • Hadoop Flags:
      Reviewed

      Description

      It would be really useful to have statistics on the number of errors and warnings in the RM and NM web UI. I'm thinking about -
      1. The number of errors and warnings in the past 5 min/1 hour/12 hours/day
      2. The top 'n'(20?) most common exceptions in the past 5 min/1 hour/12 hours/day

      By errors and warnings I'm referring to the log level.

      I suspect we can probably achieve this by writing a custom appender?(I'm open to suggestions on alternate mechanisms for implementing this).

      1. apache-yarn-2901.0.patch
        51 kB
        Varun Vasudev
      2. apache-yarn-2901.1.patch
        52 kB
        Varun Vasudev
      3. apache-yarn-2901.2.patch
        53 kB
        Varun Vasudev
      4. apache-yarn-2901.3.patch
        52 kB
        Varun Vasudev
      5. apache-yarn-2901.4.patch
        52 kB
        Varun Vasudev
      6. apache-yarn-2901.5.patch
        52 kB
        Varun Vasudev
      7. Exception collapsed.png
        340 kB
        Varun Vasudev
      8. Exception expanded.jpg
        337 kB
        Varun Vasudev
      9. Screen Shot 2015-03-19 at 7.40.02 PM.png
        508 kB
        Varun Vasudev
      10. YARN-2901.addendem.1.patch
        0.8 kB
        Wangda Tan

        Issue Links

          Activity

          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a screen shot of the what the new page looks like(for the NM). The entries that are underlined are messages that were too long(think of exception stack traces). If you click on them, it'll show the full message.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a screen shot of the what the new page looks like(for the NM). The entries that are underlined are messages that were too long(think of exception stack traces). If you click on them, it'll show the full message.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a patch with the changes.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a patch with the changes.
          Hide
          vvasudev Varun Vasudev added a comment -

          Some more information on the patch - it adds a new log appender which stores a count of the errors and warnings logged with associated time period information. It also stores the last 500 unique error and the last 500 unique warning messages as well as the number of times and the time when they occurred. Messages more than 24 hours old are purged automatically from the store. Right now the appender is added in code since log4j.properties is shared across hadoop but if it's useful, we could add it to log4j.properties itself.

          Show
          vvasudev Varun Vasudev added a comment - Some more information on the patch - it adds a new log appender which stores a count of the errors and warnings logged with associated time period information. It also stores the last 500 unique error and the last 500 unique warning messages as well as the number of times and the time when they occurred. Messages more than 24 hours old are purged automatically from the store. Right now the appender is added in code since log4j.properties is shared across hadoop but if it's useful, we could add it to log4j.properties itself.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705600/apache-yarn-2901.0.patch
          against trunk revision 522c897.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          org.apache.hadoop.yarn.server.resourcemanager.TestRM

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7023//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7023//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7023//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705600/apache-yarn-2901.0.patch against trunk revision 522c897. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation org.apache.hadoop.yarn.server.resourcemanager.TestRM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7023//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7023//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7023//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Varun Vasudev, It's a great add-on to RM web UI, with this, we don't need access RM log every time. Love the proposal

          Some suggestions about UI:
          1) Can we have a separate table to show recent X warnings/error in addition to "most frequent warnings/errors"?
          2) Is it possible to get exception stack trace as a part of the warnings/errors page. Sometimes stack trace is not a part of warn/error, but they're also very useful to troubleshooting issues.

          Show
          leftnoteasy Wangda Tan added a comment - Varun Vasudev , It's a great add-on to RM web UI, with this, we don't need access RM log every time. Love the proposal Some suggestions about UI: 1) Can we have a separate table to show recent X warnings/error in addition to "most frequent warnings/errors"? 2) Is it possible to get exception stack trace as a part of the warnings/errors page. Sometimes stack trace is not a part of warn/error, but they're also very useful to troubleshooting issues.
          Hide
          vvasudev Varun Vasudev added a comment -

          Wangda Tan

          1) Can we have a separate table to show recent X warnings/error in addition to "most frequent warnings/errors"?

          You can sort by date in the table. By default, the table is sorted to show you the most recent errors/warnings. The counts are just a collapse so that if you have the same error occurring over and over again, it doesn't fill up the table.

          2) Is it possible to get exception stack trace as a part of the warnings/errors page. Sometimes stack trace is not a part of warn/error, but they're also very useful to troubleshooting issues.

          Uploaded a screenshots of what an exception collapsed and expanded will look like. Can you let me know if it looks ok? The different colors are some issue with the exporting.

          Show
          vvasudev Varun Vasudev added a comment - Wangda Tan 1) Can we have a separate table to show recent X warnings/error in addition to "most frequent warnings/errors"? You can sort by date in the table. By default, the table is sorted to show you the most recent errors/warnings. The counts are just a collapse so that if you have the same error occurring over and over again, it doesn't fill up the table. 2) Is it possible to get exception stack trace as a part of the warnings/errors page. Sometimes stack trace is not a part of warn/error, but they're also very useful to troubleshooting issues. Uploaded a screenshots of what an exception collapsed and expanded will look like. Can you let me know if it looks ok? The different colors are some issue with the exporting.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to address the findbugs errors.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to address the findbugs errors.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705935/apache-yarn-2901.1.patch
          against trunk revision 8041267.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
          org.apache.hadoop.yarn.server.resourcemanager.TestRM

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7043//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7043//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705935/apache-yarn-2901.1.patch against trunk revision 8041267. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.TestRM Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7043//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7043//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Patched and tried, UI looks great! One suggestion is: do we need fold single line message when line is too long to be:
          Unable to load native-hadoop library for your platform... using builtin-java cla
          Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
          Actually the 2nd line can be entirely show in one line.
          I think this is caused by hard coded column width, can we make it dynamic under different environment?

          Maybe we need always not fold one line message?

          Will include implementation review in next review.

          Show
          leftnoteasy Wangda Tan added a comment - Patched and tried, UI looks great! One suggestion is: do we need fold single line message when line is too long to be: Unable to load native-hadoop library for your platform... using builtin-java cla Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Actually the 2nd line can be entirely show in one line. I think this is caused by hard coded column width, can we make it dynamic under different environment? Maybe we need always not fold one line message? Will include implementation review in next review.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Varun Vasudev,

          I spent some time take a look at Log4JMetricsAppeneder implementation (will include other modified component in next round).

          1) Log4jMetricsAppender,
          1.1 Better to place in yarn-server-common?
          1.2 If you agree above, how about put into package o.a.h.y.server.metrics (or utils)?
          1.3 Rename it to Log4jWarnErrorMetricsAppender?
          1.4 Comments about implementation:
          I think currently, implementation of cleanup can be improved, now cutoff process of message/count is basically loop all items stored, which could be inefficient (imaging if number of stored message > threshold), existing logics in the patch would lead to lots of potential stored message (tons of messages could be genereated in 5 min, which is purge message task run interval).

          If you can make the data structure to be:
          SortedMap<String, SortedMap<Long, Integer>> errors (and warnings), the outside map is sorted by value (SortedMap with smallest timestamp goes first), and inside map is sorted by key (smallest timestamp goes first), purge can happen when we add any event, it will just take at most log(N=500) time to do the purge, and no extra timer task needed.

          To make SortedMap can sort by value, one way to do that can refer to http://stackoverflow.com/questions/109383/how-to-sort-a-mapkey-value-on-the-values-in-java (first answer).

          Here, value = SortedMap<Long, Integer>>, we can sort the SortedMaps according to smallest key in each SortedMap.

          And one corner case may need to consider is, it is possible a same message can have lots of different timestamps, so we need purge the inner SortedMap too.

          To make better code readability, you can wrap the SortedMap to a inner class like MessageInfo.

          Show
          leftnoteasy Wangda Tan added a comment - Hi Varun Vasudev , I spent some time take a look at Log4JMetricsAppeneder implementation (will include other modified component in next round). 1) Log4jMetricsAppender, 1.1 Better to place in yarn-server-common? 1.2 If you agree above, how about put into package o.a.h.y.server.metrics (or utils)? 1.3 Rename it to Log4jWarnErrorMetricsAppender? 1.4 Comments about implementation: I think currently, implementation of cleanup can be improved, now cutoff process of message/count is basically loop all items stored, which could be inefficient (imaging if number of stored message > threshold), existing logics in the patch would lead to lots of potential stored message (tons of messages could be genereated in 5 min, which is purge message task run interval). If you can make the data structure to be: SortedMap<String, SortedMap<Long, Integer>> errors (and warnings), the outside map is sorted by value (SortedMap with smallest timestamp goes first), and inside map is sorted by key (smallest timestamp goes first), purge can happen when we add any event, it will just take at most log(N=500) time to do the purge, and no extra timer task needed. To make SortedMap can sort by value, one way to do that can refer to http://stackoverflow.com/questions/109383/how-to-sort-a-mapkey-value-on-the-values-in-java (first answer). Here, value = SortedMap<Long, Integer>>, we can sort the SortedMaps according to smallest key in each SortedMap. And one corner case may need to consider is, it is possible a same message can have lots of different timestamps, so we need purge the inner SortedMap too. To make better code readability, you can wrap the SortedMap to a inner class like MessageInfo.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the review Wangda Tan!

          1.1 Better to place in yarn-server-common?
          1.2 If you agree above, how about put into package o.a.h.y.server.metrics (or utils)?

          I'd prefer not to move it. All the common web ui classes(for the existing web ui) are in hadoop-yarn-common and I'll have to move everything over to hadoop-yarn-server-common.

          1.3 Rename it to Log4jWarnErrorMetricsAppender?

          Fixed.

          {cutoff}

          1.4 Comments about implementation:
          I think currently, implementation of cleanup can be improved, now cutoff process of message/count is basically loop all items stored, which could be inefficient (imaging if number of stored message > threshold), existing logics in the patch would lead to lots of potential stored message (tons of messages could be genereated in 5 min, which is purge message task run interval).

          Changed the purge implementation. I maintain a purge information structure that makes purging more efficient.

          I've also added the appender information to log4j.properties so that the appender can be enabled/disabled using YARN_ROOT_LOGGER.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the review Wangda Tan ! 1.1 Better to place in yarn-server-common? 1.2 If you agree above, how about put into package o.a.h.y.server.metrics (or utils)? I'd prefer not to move it. All the common web ui classes(for the existing web ui) are in hadoop-yarn-common and I'll have to move everything over to hadoop-yarn-server-common. 1.3 Rename it to Log4jWarnErrorMetricsAppender? Fixed. {cutoff} 1.4 Comments about implementation: I think currently, implementation of cleanup can be improved, now cutoff process of message/count is basically loop all items stored, which could be inefficient (imaging if number of stored message > threshold), existing logics in the patch would lead to lots of potential stored message (tons of messages could be genereated in 5 min, which is purge message task run interval). Changed the purge implementation. I maintain a purge information structure that makes purging more efficient. I've also added the appender information to log4j.properties so that the appender can be enabled/disabled using YARN_ROOT_LOGGER.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707723/apache-yarn-2901.2.patch
          against trunk revision af618f2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRMHA
          org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication
          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
          org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7120//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7120//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7120//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707723/apache-yarn-2901.2.patch against trunk revision af618f2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMHA org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7120//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7120//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7120//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Hi Varun,
          Thanks for updating patch, I took a look again, it's much efficient now, few comments for Log4jWarningErrorAppeneder:

          • It's better to use ReentrantReadWriteLock instead of synchronize lock, since the class will be more concurrently/frequently read comparing to write.
          • getElementsAndCounts, when it trying to get number of count for each message, it loops elements in qualifyingTimes. So in extreme case, for message="x", there's one message in every second, it will loops all 24 * 60 * 60 * 2 = 172800 items. Solution of this could be complex, either we don't have to remember every second count for some time, such as hard define we only remember #count for single message like past 1min, past 5min, past 30min, past 1h ... to avoid this problem. Or introduce tree-like structure for example interval tree (http://en.wikipedia.org/wiki/Interval_tree) to make query more efficient. I think remember limited number of time ranges for each message should be enough. There seems no need to support cases like "give me error count of 2:30:01 am to 3:40:05 pm". If you think changes are manageable, you can do it in the patch, or you can file a ticket to address in a separated patch.
          • When map.size() > maxUniqueMessages, cleanup will be triggered, I suggest to make a buffer that cleanup will not run too offen, such as when map.size() > maxUniqueMessages * 2?

          And could you take a look at findbugs warning and failed tests?

          Wangda

          Show
          leftnoteasy Wangda Tan added a comment - Hi Varun, Thanks for updating patch, I took a look again, it's much efficient now, few comments for Log4jWarningErrorAppeneder: It's better to use ReentrantReadWriteLock instead of synchronize lock, since the class will be more concurrently/frequently read comparing to write. getElementsAndCounts , when it trying to get number of count for each message, it loops elements in qualifyingTimes. So in extreme case, for message="x", there's one message in every second, it will loops all 24 * 60 * 60 * 2 = 172800 items. Solution of this could be complex, either we don't have to remember every second count for some time, such as hard define we only remember #count for single message like past 1min, past 5min, past 30min, past 1h ... to avoid this problem. Or introduce tree-like structure for example interval tree ( http://en.wikipedia.org/wiki/Interval_tree ) to make query more efficient. I think remember limited number of time ranges for each message should be enough. There seems no need to support cases like "give me error count of 2:30:01 am to 3:40:05 pm". If you think changes are manageable, you can do it in the patch, or you can file a ticket to address in a separated patch. When map.size() > maxUniqueMessages, cleanup will be triggered, I suggest to make a buffer that cleanup will not run too offen, such as when map.size() > maxUniqueMessages * 2? And could you take a look at findbugs warning and failed tests? Wangda
          Hide
          vvasudev Varun Vasudev added a comment -

          It's better to use ReentrantReadWriteLock instead of synchronize lock, since the class will be more concurrently/frequently read comparing to write.

          Actually, in this case, I suspect there will be more writes than reads. The reads will occur only when a user loads the webpage whereas the writes will occur every time an error or warning is logged. I think the synchronized lock should be ok. On a vm on my macbook pro, the appender was able to process randomly generated error messages at 150K messages per minute without any trouble.

          getElementsAndCounts, when it trying to get number of count for each message, it loops elements in qualifyingTimes. So in extreme case, for message="x", there's one message in every second, it will loops all 24 * 60 * 60 * 2 = 172800 items. Solution of this could be complex, either we don't have to remember every second count for some time, such as hard define we only remember #count for single message like past 1min, past 5min, past 30min, past 1h ... to avoid this problem. Or introduce tree-like structure for example interval tree (http://en.wikipedia.org/wiki/Interval_tree) to make query more efficient. I think remember limited number of time ranges for each message should be enough. There seems no need to support cases like "give me error count of 2:30:01 am to 3:40:05 pm". If you think changes are manageable, you can do it in the patch, or you can file a ticket to address in a separated patch.

          I think I would prefer to support the use case for arbitrary time limits for now. The summation occurs only when a user loads the web ui; so the cost shouldn't be too bad. I don't think the interval tree makes it any easier and makes the implementation much more complex.

          When map.size() > maxUniqueMessages, cleanup will be triggered, I suggest to make a buffer that cleanup will not run too offen, such as when map.size() > maxUniqueMessages * 2?

          Good point; but I think the factor of 2 is too large - it will confuse users who set the limit(since the limit will be exceeded often). I've changed it to map.size() > maxUniqueMessages * 1.1f

          And could you take a look at findbugs warning and failed tests?

          The failed tests are unrelated. I've fixed the findbugs warning.

          Show
          vvasudev Varun Vasudev added a comment - It's better to use ReentrantReadWriteLock instead of synchronize lock, since the class will be more concurrently/frequently read comparing to write. Actually, in this case, I suspect there will be more writes than reads. The reads will occur only when a user loads the webpage whereas the writes will occur every time an error or warning is logged. I think the synchronized lock should be ok. On a vm on my macbook pro, the appender was able to process randomly generated error messages at 150K messages per minute without any trouble. getElementsAndCounts, when it trying to get number of count for each message, it loops elements in qualifyingTimes. So in extreme case, for message="x", there's one message in every second, it will loops all 24 * 60 * 60 * 2 = 172800 items. Solution of this could be complex, either we don't have to remember every second count for some time, such as hard define we only remember #count for single message like past 1min, past 5min, past 30min, past 1h ... to avoid this problem. Or introduce tree-like structure for example interval tree ( http://en.wikipedia.org/wiki/Interval_tree ) to make query more efficient. I think remember limited number of time ranges for each message should be enough. There seems no need to support cases like "give me error count of 2:30:01 am to 3:40:05 pm". If you think changes are manageable, you can do it in the patch, or you can file a ticket to address in a separated patch. I think I would prefer to support the use case for arbitrary time limits for now. The summation occurs only when a user loads the web ui; so the cost shouldn't be too bad. I don't think the interval tree makes it any easier and makes the implementation much more complex. When map.size() > maxUniqueMessages, cleanup will be triggered, I suggest to make a buffer that cleanup will not run too offen, such as when map.size() > maxUniqueMessages * 2? Good point; but I think the factor of 2 is too large - it will confuse users who set the limit(since the limit will be exceeded often). I've changed it to map.size() > maxUniqueMessages * 1.1f And could you take a look at findbugs warning and failed tests? The failed tests are unrelated. I've fixed the findbugs warning.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12708332/apache-yarn-2901.3.patch
          against trunk revision 85dc3c1.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager

          The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestFileCreation

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7164//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7164//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12708332/apache-yarn-2901.3.patch against trunk revision 85dc3c1. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager The following test timeouts occurred in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestFileCreation Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7164//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7164//console This message is automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch with class marked as Private and Unstable.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch with class marked as Private and Unstable.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12708369/apache-yarn-2901.4.patch
          against trunk revision 85dc3c1.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7166//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7166//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12708369/apache-yarn-2901.4.patch against trunk revision 85dc3c1. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7166//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7166//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          On a vm on my macbook pro, the appender was able to process randomly generated error messages at 150K messages per minute without any trouble.

          Sounds good to me!

          I think I would prefer to support the use case for arbitrary time limits for now. The summation occurs only when a user loads the web ui; so the cost shouldn't be too bad. I don't think the interval tree makes it any easier and makes the implementation much more complex.

          Yeah, I don't like use too complex data structure before really needed. I agree what you said, let's keep it as-is and change it if really required.

          Good point; but I think the factor of 2 is too large - it will confuse users who set the limit(since the limit will be exceeded often). I've changed it to map.size() > maxUniqueMessages * 1.1f

          I realized if we set clean-up-threshold > maxUniqueMessages, user can see it, how about doing clean-up in two conditions:
          1) User get message, and #message > maxUniqueMessages
          2) #messages > message-threshold, we can set the message-threshold to higher to avoid too frequent cleanup.
          Sounds good?

          I'd prefer not to move it. All the common web ui classes(for the existing web ui) are in hadoop-yarn-common and I'll have to move everything over to hadoop-yarn-server-common.

          I just tried to move that, it seems no more issues happen, could you check that?

          Show
          leftnoteasy Wangda Tan added a comment - On a vm on my macbook pro, the appender was able to process randomly generated error messages at 150K messages per minute without any trouble. Sounds good to me! I think I would prefer to support the use case for arbitrary time limits for now. The summation occurs only when a user loads the web ui; so the cost shouldn't be too bad. I don't think the interval tree makes it any easier and makes the implementation much more complex. Yeah, I don't like use too complex data structure before really needed. I agree what you said, let's keep it as-is and change it if really required. Good point; but I think the factor of 2 is too large - it will confuse users who set the limit(since the limit will be exceeded often). I've changed it to map.size() > maxUniqueMessages * 1.1f I realized if we set clean-up-threshold > maxUniqueMessages, user can see it, how about doing clean-up in two conditions: 1) User get message, and #message > maxUniqueMessages 2) #messages > message-threshold, we can set the message-threshold to higher to avoid too frequent cleanup. Sounds good? I'd prefer not to move it. All the common web ui classes(for the existing web ui) are in hadoop-yarn-common and I'll have to move everything over to hadoop-yarn-server-common. I just tried to move that, it seems no more issues happen, could you check that?
          Hide
          vvasudev Varun Vasudev added a comment -

          I realized if we set clean-up-threshold > maxUniqueMessages, user can see it, how about doing clean-up in two conditions:
          1) User get message, and #message > maxUniqueMessages
          2) #messages > message-threshold, we can set the message-threshold to higher to avoid too frequent cleanup.
          Sounds good?

          Makes sense; made the change.

          I just tried to move that, it seems no more issues happen, could you check that?

          Moved ErrorAndWarningsBlock to hadoop-yarn-server-common. Renamed ErrorsAndWarningsPage in RM and NM to RMErrorsAndWarningsPage and NMErrorsAndWarningsPage.

          Show
          vvasudev Varun Vasudev added a comment - I realized if we set clean-up-threshold > maxUniqueMessages, user can see it, how about doing clean-up in two conditions: 1) User get message, and #message > maxUniqueMessages 2) #messages > message-threshold, we can set the message-threshold to higher to avoid too frequent cleanup. Sounds good? Makes sense; made the change. I just tried to move that, it seems no more issues happen, could you check that? Moved ErrorAndWarningsBlock to hadoop-yarn-server-common. Renamed ErrorsAndWarningsPage in RM and NM to RMErrorsAndWarningsPage and NMErrorsAndWarningsPage.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12709021/apache-yarn-2901.5.patch
          against trunk revision 9ed43f2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7199//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7199//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12709021/apache-yarn-2901.5.patch against trunk revision 9ed43f2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7199//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7199//console This message is automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          +1 for the patch. Will commit it today if no opposite opinions.

          Show
          leftnoteasy Wangda Tan added a comment - +1 for the patch. Will commit it today if no opposite opinions.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed to trunk/branch-2, thanks Varun Vasudev!

          Show
          leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Varun Vasudev !
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7501 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7501/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7501 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7501/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #152 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/152/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #152 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/152/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #886 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/886/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #886 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/886/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #143 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/143/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #143 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/143/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2084 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2084/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2084 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2084/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #153 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/153/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #153 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/153/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2102/)
          YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb)

          • hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2102/ ) YARN-2901 . Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) (wangda: rev bad070fe15a642cc6f3a165612fbd272187e03cb) hadoop-common-project/hadoop-common/src/main/conf/log4j.properties hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLog4jWarningErrorMetricsAppender.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/ErrorsAndWarningsBlock.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMErrorsAndWarningsPage.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Log4jWarningErrorMetricsAppender.java
          Hide
          gtCarrera9 Li Lu added a comment -

          The recent commit added two findbugs warnings on my local machine and in the build of YARN-3426:

          
          Code	Warning
          UrF	Unread public/protected field: org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element.count
          UrF	Unread public/protected field: org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element.timestampSeconds
          

          From git blame this JIRA is the one performed the most recent change. Reopen this JIRA to fix them.

          Show
          gtCarrera9 Li Lu added a comment - The recent commit added two findbugs warnings on my local machine and in the build of YARN-3426 : Code Warning UrF Unread public / protected field: org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element.count UrF Unread public / protected field: org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element.timestampSeconds From git blame this JIRA is the one performed the most recent change. Reopen this JIRA to fix them.
          Hide
          gtCarrera9 Li Lu added a comment -

          BTW, Jenkins didn't report those two warnings in this JIRA because probably it ran against another patch in the run 4 days ago.

          Show
          gtCarrera9 Li Lu added a comment - BTW, Jenkins didn't report those two warnings in this JIRA because probably it ran against another patch in the run 4 days ago.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks for reporting, Li Lu. This is false alarm from findbugs. Fields of org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element are used by ErrosAndWarningsBlock. Simply exclude such warnings.

          Uploaded addendum patch and pending Jenkins.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks for reporting, Li Lu . This is false alarm from findbugs. Fields of org.apache.hadoop.yarn.util.Log4jWarningErrorMetricsAppender$Element are used by ErrosAndWarningsBlock. Simply exclude such warnings. Uploaded addendum patch and pending Jenkins.
          Hide
          gtCarrera9 Li Lu added a comment -

          +1 pending Jenkins.

          Show
          gtCarrera9 Li Lu added a comment - +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12723490/YARN-2901.addendem.1.patch
          against trunk revision 3fb5abf.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7230//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7230//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723490/YARN-2901.addendem.1.patch against trunk revision 3fb5abf. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7230//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7230//console This message is automatically generated.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          IMHO, using calls to LOG.error()/LOG.warn() as proxies for counting errors/warnings is flaky at best. It places cumbersome restrictions on code requiring that a given error/warning correspond to a single error()/warn() call. This is tough to enforce even within a single block of code, let alone across multiple functions ( e.g when an exception is thrown/re-thrown and an error/warning logged in multiple locations ). I hope this will not lead to a restriction on new code in YARN that error/warning should corresponding a single error()/warn() call.

          Show
          sidharta-s Sidharta Seethana added a comment - IMHO, using calls to LOG.error()/LOG.warn() as proxies for counting errors/warnings is flaky at best. It places cumbersome restrictions on code requiring that a given error/warning correspond to a single error()/warn() call. This is tough to enforce even within a single block of code, let alone across multiple functions ( e.g when an exception is thrown/re-thrown and an error/warning logged in multiple locations ). I hope this will not lead to a restriction on new code in YARN that error/warning should corresponding a single error()/warn() call.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Committed addendum patch fixed findbugs warnings to trunk/branch-2.

          Show
          leftnoteasy Wangda Tan added a comment - Committed addendum patch fixed findbugs warnings to trunk/branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7534 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7534/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7534 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7534/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/158/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #158 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/158/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2090 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2090/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2090 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2090/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/149/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #149 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/149/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #892 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/892/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #892 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/892/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #159 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/159/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #159 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/159/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2108/)
          YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0)

          • hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2108 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2108/ ) YARN-2901 addendum: Fixed findbugs warning caused by previously patch (wangda: rev ba9ee22ca4ed2c5ff447b66b2e2dfe25f6880fe0) hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml

            People

            • Assignee:
              vvasudev Varun Vasudev
              Reporter:
              vvasudev Varun Vasudev
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development