HBase
  1. HBase
  2. HBASE-4709

Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.90.4, 0.92.0, 0.94.0, 0.94.1, 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Since switching over HBase to build with Hadoop 0.20.205.0, we've been getting a lot of metrics related errors in the log files for tests:

      2011-10-30 22:00:22,858 INFO  [main] log.Slf4jLog(67): jetty-6.1.26
      2011-10-30 22:00:22,871 INFO  [main] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode to /tmp/Jetty_localhost_55751_datanode____.kw16hy/webapp
      2011-10-30 22:00:23,048 INFO  [main] log.Slf4jLog(67): Started SelectChannelConnector@localhost:55751
      Starting DataNode 1 with dfs.data.dir: /home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
      2011-10-30 22:00:23,237 WARN  [main] impl.MetricsSystemImpl(137): Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
      2011-10-30 22:00:23,237 WARN  [main] util.MBeans(59): Hadoop:service=DataNode,name=MetricsSystem,sub=Control
      javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
      	at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
      	at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
      	at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
      	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
      	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
      	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
      	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
      	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
      	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
      	at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
      

      This seems to be due to errors initializing the new hadoop metrics2 code by default, when running in a mini cluster. The errors themselves seem to be harmless – they're not breaking any tests – but we should figure out what configuration we need to eliminate them.

      1. 4709.v2.patch
        2 kB
        Nicolas Liochon
      2. 4709.v2.patch
        2 kB
        Nicolas Liochon
      3. 4709_workaround.v1.patch
        1 kB
        Nicolas Liochon

        Issue Links

          Activity

          Hide
          Lewis John McGibbney added a comment -

          We have the exact same issues over at Apache Gora. As you mention, the build is not failing but there are many metrics WARN instances, with identical output. I came across this when looking for a solution. I can confirm that this is replicated on hadoop-core 1.0.1, hadoop-tests 1.0.1 and hbase 0.90.4 core and tests.

          Show
          Lewis John McGibbney added a comment - We have the exact same issues over at Apache Gora. As you mention, the build is not failing but there are many metrics WARN instances, with identical output. I came across this when looking for a solution. I can confirm that this is replicated on hadoop-core 1.0.1, hadoop-tests 1.0.1 and hbase 0.90.4 core and tests.
          Hide
          stack added a comment -

          This particular complaint is about namenode trying to register same bean twice which seems a little odd:

          InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
          
          Show
          stack added a comment - This particular complaint is about namenode trying to register same bean twice which seems a little odd: InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
          Hide
          Lewis John McGibbney added a comment -

          An example of our typical stack trace.

          3287 [main] INFO org.mortbay.log - jetty-6.1.26
          3333 [main] INFO org.mortbay.log - Extract jar:file:/export/home/hudson/hudson-slave/maven-repositories/0/org/apache/hadoop/hadoop-core/1.0.1/hadoop-core-1.0.1.jar!/webapps/hdfs to /var/tmp/Jetty_localhost_54201_hdfs____txys7y/webapp
          3598 [main] INFO org.mortbay.log - Started SelectChannelConnector@localhost:54201
          12/04/05 16:56:06 INFO namenode.NameNode: Web-server up at: localhost:54201
          12/04/05 16:56:06 INFO ipc.Server: IPC Server Responder: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server listener on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 0 on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 3 on 54200: starting
          12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 7 on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 2 on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 1 on 54200: starting
          12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 9 on 54200: starting
          12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 8 on 54200: starting
          12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 6 on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 4 on 54200: starting
          12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 5 on 54200: starting
          Starting DataNode 0 with dfs.data.dir: /export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data1,/export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data2
          12/04/05 16:56:07 WARN impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
          12/04/05 16:56:07 WARN util.MBeans: Hadoop:service=DataNode,name=MetricsSystem,sub=Control
          javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
          	at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
          	at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
          	at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
          	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
          	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
          	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
          	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
          	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
          	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
          	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
          	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
          	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
          	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1520)
          	at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
          	at org.apache.hadoop.mapred.HadoopTestCase.setUp(HadoopTestCase.java:148)
          	at org.apache.gora.mapreduce.DataStoreMapReduceTestBase.setUp(DataStoreMapReduceTestBase.java:61)
          	at junit.framework.TestCase.runBare(TestCase.java:132)
          	at junit.framework.TestResult$1.protect(TestResult.java:110)
          	at junit.framework.TestResult.runProtected(TestResult.java:128)
          	at junit.framework.TestResult.run(TestResult.java:113)
          	at junit.framework.TestCase.run(TestCase.java:124)
          	at junit.framework.TestSuite.runTest(TestSuite.java:243)
          	at junit.framework.TestSuite.run(TestSuite.java:238)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
          	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
          	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
          

          You will however notice a difference between the original and my trace, where this particular one is not using HBaseTestingUtility, however instead relying on HadoopTestCase which in turn invokes MiniDFSCluster... the problem to that end is identical. I must state that there is ONE previous 'metrics system not starting' warning identical to the one you see above. E.g. two in all the 2nd being followed by the full stack trace.

          Show
          Lewis John McGibbney added a comment - An example of our typical stack trace. 3287 [main] INFO org.mortbay.log - jetty-6.1.26 3333 [main] INFO org.mortbay.log - Extract jar:file:/export/home/hudson/hudson-slave/maven-repositories/0/org/apache/hadoop/hadoop-core/1.0.1/hadoop-core-1.0.1.jar!/webapps/hdfs to / var /tmp/Jetty_localhost_54201_hdfs____txys7y/webapp 3598 [main] INFO org.mortbay.log - Started SelectChannelConnector@localhost:54201 12/04/05 16:56:06 INFO namenode.NameNode: Web-server up at: localhost:54201 12/04/05 16:56:06 INFO ipc.Server: IPC Server Responder: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server listener on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 0 on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 3 on 54200: starting 12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 7 on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 2 on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 1 on 54200: starting 12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 9 on 54200: starting 12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 8 on 54200: starting 12/04/05 16:56:07 INFO ipc.Server: IPC Server handler 6 on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 4 on 54200: starting 12/04/05 16:56:06 INFO ipc.Server: IPC Server handler 5 on 54200: starting Starting DataNode 0 with dfs.data.dir: /export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data1,/export/home/hudson/hudson-slave/workspace/gora-trunk/trunk./gora-core/target/test-data/dfs/data/data2 12/04/05 16:56:07 WARN impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-datanode.properties, hadoop-metrics2.properties 12/04/05 16:56:07 WARN util.MBeans: Hadoop:service=DataNode,name=MetricsSystem,sub=Control javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control at com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120) at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143) at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482) at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1520) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124) at org.apache.hadoop.mapred.HadoopTestCase.setUp(HadoopTestCase.java:148) at org.apache.gora.mapreduce.DataStoreMapReduceTestBase.setUp(DataStoreMapReduceTestBase.java:61) at junit.framework.TestCase.runBare(TestCase.java:132) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) You will however notice a difference between the original and my trace, where this particular one is not using HBaseTestingUtility, however instead relying on HadoopTestCase which in turn invokes MiniDFSCluster... the problem to that end is identical. I must state that there is ONE previous 'metrics system not starting' warning identical to the one you see above. E.g. two in all the 2nd being followed by the full stack trace.
          Hide
          merve temizer added a comment -

          The issue does not allow me to build and test Apache James.

          Show
          merve temizer added a comment - The issue does not allow me to build and test Apache James.
          Hide
          stack added a comment -

          Its just a WARN message? Is it preventing JAMEs coming up?

          Show
          stack added a comment - Its just a WARN message? Is it preventing JAMEs coming up?
          Hide
          merve temizer added a comment -

          Yes it is preventing JAMEs coming up.

          Maybe because of me that do not change anything in JAMES and only try

          mvn clean package

          or

          mvn clean install

          Should i try to edit something to by pass HBase test?

          In issue page someone tells it does not prevent James to come up.

          Show
          merve temizer added a comment - Yes it is preventing JAMEs coming up. Maybe because of me that do not change anything in JAMES and only try mvn clean package or mvn clean install Should i try to edit something to by pass HBase test? In issue page someone tells it does not prevent James to come up.
          Hide
          stack added a comment -

          In issue page someone tells it does not prevent James to come up.

          Can you tell us more why the above WARN message is preventing James launching? Maybe its not coming up for some other reason? You have more logs for us to look at? Ask on dev list and add log there rather than here?

          Show
          stack added a comment - In issue page someone tells it does not prevent James to come up. Can you tell us more why the above WARN message is preventing James launching? Maybe its not coming up for some other reason? You have more logs for us to look at? Ask on dev list and add log there rather than here?
          Hide
          Nicolas Liochon added a comment -

          A workaround, proposed on the mailing list Friday 23th November.

          Show
          Nicolas Liochon added a comment - A workaround, proposed on the mailing list Friday 23th November.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554834/4709_workaround.v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          -1 javadoc. The javadoc tool appears to have generated 98 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestHCM

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554834/4709_workaround.v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. -1 javadoc . The javadoc tool appears to have generated 98 warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 24 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3400//console This message is automatically generated.
          Hide
          stack added a comment -

          +1 if the TestHCM is not because of this patch. Thanks for doing this N.

          Show
          stack added a comment - +1 if the TestHCM is not because of this patch. Thanks for doing this N.
          Hide
          Elliott Clark added a comment -

          Should the static logger definitions be moved into hbase-hadoop

          {1|2}

          -compat? Though right now it works because hadoop 1 and 2 both happen to have that class in the same location.

          Show
          Elliott Clark added a comment - Should the static logger definitions be moved into hbase-hadoop {1|2} -compat? Though right now it works because hadoop 1 and 2 both happen to have that class in the same location.
          Hide
          Nicolas Liochon added a comment -

          Sorry for the delay, I haven't seen the comments.
          Elliott, I don't understand your proposition? May be it's unrelated, but I though having it in test only was useful because it's a workaround useful for tests only: there is no reason to have this workaround in production.

          Show
          Nicolas Liochon added a comment - Sorry for the delay, I haven't seen the comments. Elliott, I don't understand your proposition? May be it's unrelated, but I though having it in test only was useful because it's a workaround useful for tests only: there is no reason to have this workaround in production.
          Hide
          Elliott Clark added a comment -

          So I had two thoughts.

          1. I'm not 100% sure that this kind of work around isn't needed in production. Right now lots of messages are logged by the metrics system that are really scary to users but not really a bad thing (Specifically when starting/stopping the metrics system there are tons of errors thrown). So we could want this logging change for more than just one logger.
          2. If we decide that we shouldn't silence things in a real server there are already test utilities that reach into Hadoop's guts that are in hbase-hadoop1-compat and hbase-hadoop2-compat. Seems like we should try and make the main jar have very few places where Hadoop internals are touched just in case different versions diverge more in the future.
          Show
          Elliott Clark added a comment - So I had two thoughts. I'm not 100% sure that this kind of work around isn't needed in production. Right now lots of messages are logged by the metrics system that are really scary to users but not really a bad thing (Specifically when starting/stopping the metrics system there are tons of errors thrown). So we could want this logging change for more than just one logger. If we decide that we shouldn't silence things in a real server there are already test utilities that reach into Hadoop's guts that are in hbase-hadoop1-compat and hbase-hadoop2-compat. Seems like we should try and make the main jar have very few places where Hadoop internals are touched just in case different versions diverge more in the future.
          Hide
          Sergey Shelukhin added a comment -

          Is this issue to fix the error, or get rid of the message? +1 to get rid of the message even if the error is acceptable, it clogs logs. Also "Illegal stop" exception dump when clearing JMX cache...

          Show
          Sergey Shelukhin added a comment - Is this issue to fix the error, or get rid of the message? +1 to get rid of the message even if the error is acceptable, it clogs logs. Also "Illegal stop" exception dump when clearing JMX cache...
          Hide
          Elliott Clark added a comment -

          It's the same way to fix both log spams, so we should fix both at the same time if it's thought to be a good idea.

          Show
          Elliott Clark added a comment - It's the same way to fix both log spams, so we should fix both at the same time if it's thought to be a good idea.
          Hide
          Nicolas Liochon added a comment -

          Ok, then we could also put them in the default log4j property (in conf) + the HBaseTestingUtility for people linking directly with it for a reason or another? Or you think we should force the values anyway?

          Show
          Nicolas Liochon added a comment - Ok, then we could also put them in the default log4j property (in conf) + the HBaseTestingUtility for people linking directly with it for a reason or another? Or you think we should force the values anyway?
          Hide
          Elliott Clark added a comment -

          My thought was putting the defaults in the log4j conf + in both of the MetricsAssertHelpers.

          Show
          Elliott Clark added a comment - My thought was putting the defaults in the log4j conf + in both of the MetricsAssertHelpers.
          Hide
          Nicolas Liochon added a comment -

          I've done the log4j & HBaseTestingUtility in v2. Note sure about what to do in MetricsAssertHelper, but may be I can commit this part first? Waiting for a +1 before doing so.

          Show
          Nicolas Liochon added a comment - I've done the log4j & HBaseTestingUtility in v2. Note sure about what to do in MetricsAssertHelper, but may be I can commit this part first? Waiting for a +1 before doing so.
          Hide
          Ted Yu added a comment -

          Verified that the following no longer appears in test output after applying this patch:

          target/surefire-reports/TEST-org.apache.hadoop.hbase.replication.TestMasterReplication.xml:javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
          
          Show
          Ted Yu added a comment - Verified that the following no longer appears in test output after applying this patch: target/surefire-reports/TEST-org.apache.hadoop.hbase.replication.TestMasterReplication.xml:javax.management.InstanceAlreadyExistsException: MXBean already registered with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
          Hide
          Nicolas Liochon added a comment -

          V2 committed.

          Show
          Nicolas Liochon added a comment - V2 committed.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #3603 (See https://builds.apache.org/job/HBase-TRUNK/3603/)
          HBASE-4709 Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors (Revision 1419296)

          Result = FAILURE
          nkeywal :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/resources/log4j.properties
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #3603 (See https://builds.apache.org/job/HBase-TRUNK/3603/ ) HBASE-4709 Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors (Revision 1419296) Result = FAILURE nkeywal : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/resources/log4j.properties
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #290 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/290/)
          HBASE-4709 Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors (Revision 1419296)

          Result = FAILURE
          nkeywal :
          Files :

          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/hbase-server/src/test/resources/log4j.properties
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #290 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/290/ ) HBASE-4709 Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors (Revision 1419296) Result = FAILURE nkeywal : Files : /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/hbase-server/src/test/resources/log4j.properties
          Hide
          Lewis John McGibbney added a comment -

          Hi, I've been watching the thread with interest. Can I please confirm that this will be integrated into 0.96.0. Correct?

          Show
          Lewis John McGibbney added a comment - Hi, I've been watching the thread with interest. Can I please confirm that this will be integrated into 0.96.0. Correct?
          Hide
          Nicolas Liochon added a comment -

          Yes, the file 4709.v2.patch is committed to trunk, so the 0.96 will have it.

          Show
          Nicolas Liochon added a comment - Yes, the file 4709.v2.patch is committed to trunk, so the 0.96 will have it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12559861/4709.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3524//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12559861/4709.v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3524//console This message is automatically generated.
          Hide
          stack added a comment -

          This was committed to trunk a while back. Resolving.

          Show
          stack added a comment - This was committed to trunk a while back. Resolving.
          Hide
          Ted Yu added a comment -
          Show
          Ted Yu added a comment - InstanceAlreadyExistsException appears in 0.94 test output as well e.g. https://builds.apache.org/job/HBase-0.94/776/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Shall we port the fix to 0.94 ?
          Hide
          stack added a comment -

          Marking closed.

          Show
          stack added a comment - Marking closed.

            People

            • Assignee:
              Unassigned
              Reporter:
              Gary Helmling
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development