Hadoop Common
  1. Hadoop Common
  2. HADOOP-8027

Visiting /jmx on the daemon web interfaces may print unnecessary error in logs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 1.0.3, 0.23.1, 0.24.0
    • Component/s: metrics
    • Labels:
      None

      Description

      Logs that follow a /jmx servlet visit:

      11/11/22 12:09:52 ERROR jmx.JMXJsonServlet: getting attribute UsageThreshold of java.lang:type=MemoryPool,name=Par Eden Space threw an exception
      javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
      ...
      
      1. HDFS-2584.patch
        2 kB
        Aaron T. Myers
      2. HADOOP-8027.branch-1.patch
        1 kB
        Hitesh Shah

        Activity

        Hide
        Chris Leroy added a comment -

        I think the exception being thrown is not a problem. We're getting it because in exploring the JMX bean the way we are we're effectively trying to get the usage threshold of a memory pool where isUsageThresholdSupported is false. This leads the UnsupportedOperationException to be thrown, and then we spew. Isn't it reasonable to just catch the exception and not log when this happens?

        Something like:

        diff --git a/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java b/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java
        index 2c8f797..e9d1f9e 100644
        — a/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java
        +++ b/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java
        @@ -34,6 +34,7 @@ import javax.management.MBeanServer;
        import javax.management.MalformedObjectNameException;
        import javax.management.ObjectName;
        import javax.management.ReflectionException;
        +import javax.management.RuntimeMBeanException;
        import javax.management.openmbean.CompositeData;
        import javax.management.openmbean.CompositeType;
        import javax.management.openmbean.TabularData;
        @@ -239,6 +240,15 @@ public class JMXJsonServlet extends HttpServlet

        { // and fall back on the class name LOG.error("getting attribute " + prs + " of " + oname + " threw an exception", e); + }

        catch (RuntimeMBeanException e) {
        + // The code inside the attribute getter threw an exception, so we
        + // skip outputting the attribute. We will log the exception in
        + // certain cases, but suppress the log message in others.
        + if (!(e.getCause() instanceof UnsupportedOperationException))

        { + LOG.error("getting attribute " + attName + " of " + oname + + " threw an exception", e); + }

        + return;
        } catch (RuntimeException e) {
        // For some reason even with an MBeanException available to them
        // Runtime exceptionscan still find their way through, so treat them

        Show
        Chris Leroy added a comment - I think the exception being thrown is not a problem. We're getting it because in exploring the JMX bean the way we are we're effectively trying to get the usage threshold of a memory pool where isUsageThresholdSupported is false. This leads the UnsupportedOperationException to be thrown, and then we spew. Isn't it reasonable to just catch the exception and not log when this happens? Something like: diff --git a/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java b/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java index 2c8f797..e9d1f9e 100644 — a/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java +++ b/src/core/org/apache/hadoop/jmx/JMXJsonServlet.java @@ -34,6 +34,7 @@ import javax.management.MBeanServer; import javax.management.MalformedObjectNameException; import javax.management.ObjectName; import javax.management.ReflectionException; +import javax.management.RuntimeMBeanException; import javax.management.openmbean.CompositeData; import javax.management.openmbean.CompositeType; import javax.management.openmbean.TabularData; @@ -239,6 +240,15 @@ public class JMXJsonServlet extends HttpServlet { // and fall back on the class name LOG.error("getting attribute " + prs + " of " + oname + " threw an exception", e); + } catch (RuntimeMBeanException e) { + // The code inside the attribute getter threw an exception, so we + // skip outputting the attribute. We will log the exception in + // certain cases, but suppress the log message in others. + if (!(e.getCause() instanceof UnsupportedOperationException)) { + LOG.error("getting attribute " + attName + " of " + oname + + " threw an exception", e); + } + return; } catch (RuntimeException e) { // For some reason even with an MBeanException available to them // Runtime exceptionscan still find their way through, so treat them
        Hide
        Aaron T. Myers added a comment -

        I agree with Chris's assessment - this exception will be thrown under normal conditions, and isn't indicative of anything wrong, so we might as well not log it at error level.

        Here's a patch which addresses the issue. No tests are included since we're just changing some log output. I manually verified that the exception is no longer printed to the NN log unless DEBUG output is enabled.

        Show
        Aaron T. Myers added a comment - I agree with Chris's assessment - this exception will be thrown under normal conditions, and isn't indicative of anything wrong, so we might as well not log it at error level. Here's a patch which addresses the issue. No tests are included since we're just changing some log output. I manually verified that the exception is no longer printed to the NN log unless DEBUG output is enabled.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513542/HDFS-2584.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1850//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513542/HDFS-2584.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1850//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Patch application failed because this JIRA was filed under HDFS, though all the changes are in Common. I'll go kick Hudson to run test-patch again now.

        Show
        Aaron T. Myers added a comment - Patch application failed because this JIRA was filed under HDFS, though all the changes are in Common. I'll go kick Hudson to run test-patch again now.
        Hide
        Eli Collins added a comment -

        +1 pending jenkins

        Show
        Eli Collins added a comment - +1 pending jenkins
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12513542/HDFS-2584.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/568//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/568//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513542/HDFS-2584.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/568//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/568//console This message is automatically generated.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1751/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1751 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1751/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for for the quick review, Eli. I've just committed this to trunk and the 0.23 branch.

        Show
        Aaron T. Myers added a comment - Thanks a lot for for the quick review, Eli. I've just committed this to trunk and the 0.23 branch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1678 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1678/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1678 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1678/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #495 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/495/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #495 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/495/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #505 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/505/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #505 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/505/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1691/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1691/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #513 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/513/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #513 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/513/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #949 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/949/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #162 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/162/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #184 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/184/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241308 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/)
        HADOOP-8027. Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers.

        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #982 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/982/ ) HADOOP-8027 . Visiting /jmx on the daemon web interfaces may print unnecessary error in logs. Contributed by Aaron T. Myers. atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241303 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java
        Hide
        Hitesh Shah added a comment -

        Attaching a patch for branch-1.

        Show
        Hitesh Shah added a comment - Attaching a patch for branch-1.
        Hide
        Matt Foley added a comment -

        +1. Accepted for 1.0.3. Committing to branch-1 and branch-1.0.
        Thanks, Hitesh!

        Show
        Matt Foley added a comment - +1. Accepted for 1.0.3. Committing to branch-1 and branch-1.0. Thanks, Hitesh!
        Hide
        Matt Foley added a comment -

        Of course it was actually ported to branch-1, not "hadoop 1". Sorry for the nomenclature error.

        Show
        Matt Foley added a comment - Of course it was actually ported to branch-1, not "hadoop 1". Sorry for the nomenclature error.

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development