Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      hadoop-yarn-common uses log4j directly and when trying to redirect that through an SLF4J bridge version 1.7.5 has issues, due to use of AppenderSkeleton which is missing in log4j-over-slf4j version 1.7.5.

      This is documented on the 1.7.6 release notes but 1.7.7 should be suitable.

      This is applicable to all the projects using Hadoop motherpom, but Yarn appears to be bringing Log4J in, rather than coding to the SLF4J API.

      The issue shows in the logs as follows in Yarn MR apps, which is painful to diagnose.

      WARN  [2014-11-18 09:58:06,390+0100] [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Caught exception in callback postStart
      java.lang.reflect.InvocationTargetException: null
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_71]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290) ~[job.jar:0.22-SNAPSHOT]
      	at com.sun.proxy.$Proxy2.postStart(Unknown Source) [na:na]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:157) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1036) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478) [job.jar:0.22-SNAPSHOT]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_71]
      	at javax.security.auth.Subject.doAs(Subject.java:415) [na:1.7.0_71]
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407) [job.jar:0.22-SNAPSHOT]
      Caused by: java.lang.IncompatibleClassChangeError: Implementing class
      	at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:800) ~[na:1.7.0_71]
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.7.0_71]
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) ~[na:1.7.0_71]
      	at java.net.URLClassLoader.access$100(URLClassLoader.java:71) ~[na:1.7.0_71]
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ~[na:1.7.0_71]
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_71]
      	at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_71]
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_71]
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_71]
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.source.JvmMetrics.getEventCounters(JvmMetrics.java:183) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.source.JvmMetrics.getMetrics(JvmMetrics.java:100) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151) ~[job.jar:0.22-SNAPSHOT]
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333) ~[na:1.7.0_71]
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319) ~[na:1.7.0_71]
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[na:1.7.0_71]
      	at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96) ~[job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245) [job.jar:0.22-SNAPSHOT]
      	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229) ~[job.jar:0.22-SNAPSHOT]
      	... 18 common frames omitted
      

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Linking to HADOOP-11317 to cover project-wide use.
          I don't think yarn-common needs to explicitly declare a dependency on log4j, at least outside the test run. If you comment out that dependency —does everything still build?

          Show
          stevel@apache.org Steve Loughran added a comment - Linking to HADOOP-11317 to cover project-wide use. I don't think yarn-common needs to explicitly declare a dependency on log4j, at least outside the test run. If you comment out that dependency —does everything still build?
          Hide
          timrobertson100 Tim Robertson added a comment -

          Sadly no.

          It is used in the ContainerLogAppender and ContainerRollingLogAppender.

          I tried to remove it and compile using the log4j-over-slf4j v1.7.7 bridge but that fails because the SLF4J classes are not the same API. For example the SLF4J RollingFileAppender does not implement methods like setFile(), setAppend() etc. The build will fail with the following:

          [INFO] -------------------------------------------------------------
          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[41,6] error: cannot find symbol
          [ERROR]   symbol:   method setFile(String)
            location: class ContainerRollingLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[42,6] error: cannot find symbol
          [ERROR]   symbol:   method setAppend(boolean)
            location: class ContainerRollingLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[43,11] error: cannot find symbol
          [ERROR]   symbol: method activateOptions()
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[38,2] error: method does not override or implement a method from a supertype
          [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[49,8] error: cannot find symbol
          [ERROR]   symbol:   variable qw
            location: class ContainerRollingLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[50,6] error: cannot find symbol
          [ERROR]   symbol:   variable qw
            location: class ContainerRollingLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[37,7] error: no suitable constructor found for FileAppender()
          [ERROR]     constructor FileAppender.FileAppender(Layout,String,boolean,boolean,int) is not applicable
                (actual and formal argument lists differ in length)
              constructor FileAppender.FileAppender(Layout,String,boolean) is not applicable
                (actual and formal argument lists differ in length)
              constructor FileAppender.FileAppender(Layout,String) is not applicable
                (actual and formal argument lists differ in length)
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[52,6] error: cannot find symbol
          [ERROR]   symbol:   method setFile(String)
            location: class ContainerLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[53,6] error: cannot find symbol
          [ERROR]   symbol:   method setAppend(boolean)
            location: class ContainerLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[65,13] error: cannot find symbol
          [ERROR]   symbol: method append(LoggingEvent)
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[58,2] error: method does not override or implement a method from a supertype
          [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[77,8] error: cannot find symbol
          [ERROR]   symbol:   variable qw
            location: class ContainerLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[78,6] error: cannot find symbol
          [ERROR]   symbol:   variable qw
            location: class ContainerLogAppender
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[87,13] error: cannot find symbol
          [ERROR]   symbol: method append(LoggingEvent)
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[90,9] error: cannot find symbol
          [ERROR]   symbol: method close()
          /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[82,2] error: method does not override or implement a method from a supertype
          [INFO] 16 errors 
          [INFO] -------------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          
          Show
          timrobertson100 Tim Robertson added a comment - Sadly no. It is used in the ContainerLogAppender and ContainerRollingLogAppender . I tried to remove it and compile using the log4j-over-slf4j v1.7.7 bridge but that fails because the SLF4J classes are not the same API. For example the SLF4J RollingFileAppender does not implement methods like setFile(), setAppend() etc. The build will fail with the following: [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[41,6] error: cannot find symbol [ERROR] symbol: method setFile( String ) location: class ContainerRollingLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[42,6] error: cannot find symbol [ERROR] symbol: method setAppend( boolean ) location: class ContainerRollingLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[43,11] error: cannot find symbol [ERROR] symbol: method activateOptions() /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[38,2] error: method does not override or implement a method from a supertype [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[49,8] error: cannot find symbol [ERROR] symbol: variable qw location: class ContainerRollingLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerRollingLogAppender.java:[50,6] error: cannot find symbol [ERROR] symbol: variable qw location: class ContainerRollingLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[37,7] error: no suitable constructor found for FileAppender() [ERROR] constructor FileAppender.FileAppender(Layout, String , boolean , boolean , int ) is not applicable (actual and formal argument lists differ in length) constructor FileAppender.FileAppender(Layout, String , boolean ) is not applicable (actual and formal argument lists differ in length) constructor FileAppender.FileAppender(Layout, String ) is not applicable (actual and formal argument lists differ in length) /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[52,6] error: cannot find symbol [ERROR] symbol: method setFile( String ) location: class ContainerLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[53,6] error: cannot find symbol [ERROR] symbol: method setAppend( boolean ) location: class ContainerLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[65,13] error: cannot find symbol [ERROR] symbol: method append(LoggingEvent) /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[58,2] error: method does not override or implement a method from a supertype [ERROR] /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[77,8] error: cannot find symbol [ERROR] symbol: variable qw location: class ContainerLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[78,6] error: cannot find symbol [ERROR] symbol: variable qw location: class ContainerLogAppender /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[87,13] error: cannot find symbol [ERROR] symbol: method append(LoggingEvent) /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[90,9] error: cannot find symbol [ERROR] symbol: method close() /Users/tim/dev/git/hadoop/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ContainerLogAppender.java:[82,2] error: method does not override or implement a method from a supertype [INFO] 16 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tim, add a patch in HADOOP-11317 to increment the SLF4J version and I'll apply it

          Show
          stevel@apache.org Steve Loughran added a comment - tim, add a patch in HADOOP-11317 to increment the SLF4J version and I'll apply it
          Hide
          timrobertson100 Tim Robertson added a comment -

          Done - bumped to 1.7.10 which the release notes suggest should be fine.

          Show
          timrobertson100 Tim Robertson added a comment - Done - bumped to 1.7.10 which the release notes suggest should be fine.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Resolved this since HADOOP-11317 bumped to 1.7.10

          Show
          leftnoteasy Wangda Tan added a comment - Resolved this since HADOOP-11317 bumped to 1.7.10

            People

            • Assignee:
              Unassigned
              Reporter:
              timrobertson100 Tim Robertson
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development