Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      hadoop-yarn-common uses log4j directly and when trying to redirect that through an SLF4J bridge version 1.7.5 has issues, due to use of AppenderSkeleton which is missing in log4j-over-slf4j version 1.7.5.

      This is documented on the 1.7.6 release notes but 1.7.7 should be suitable.

      This is applicable to all the projects using Hadoop motherpom, but Yarn appears to be bringing Log4J in, rather than coding to the SLF4J API.

      The issue shows in the logs as follows in Yarn MR apps, which is painful to diagnose.

      WARN  [2014-11-18 09:58:06,390+0100] [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Caught exception in callback postStart
      java.lang.reflect.InvocationTargetException: null
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290) ~[job.jar:0.22-SNAPSHOT]
      	at com.sun.proxy.$Proxy2.postStart(Unknown Source) [na:na]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:157) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1036) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478) [job.jar:0.22-SNAPSHOT]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_71]
      	at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407) [job.jar:0.22-SNAPSHOT]
      Caused by: java.lang.IncompatibleClassChangeError: Implementing class
      	at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:800) ~[na:1.7.0_71]
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.7.0_71]
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) ~[na:1.7.0_71]
      	at java.net.URLClassLoader.access$100(URLClassLoader.java:71) ~[na:1.7.0_71]
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.7.0_71]
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_71]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_71]
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_71]
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.source.JvmMetrics.getEventCounters(JvmMetrics.java:183) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.source.JvmMetrics.getMetrics(JvmMetrics.java:100) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151) ~[job.jar:0.22-SNAPSHOT]
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333) ~[na:1.7.0_71]
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319) ~[na:1.7.0_71]
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229) ~[job.jar:0.22-SNAPSHOT]
      	... 18 common frames omitted
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                timrobertson100 Tim Robertson
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: