Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3129

Fix the joda library conflicts during Kylin start on EMR 5.8+

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Workaround
    • Affects Version/s: None
    • Fix Version/s: v2.3.0
    • Component/s: Environment
    • Labels:
      None

      Description

      When run "bin/kylin.sh start", it reports an NoSuchMethodError:

      Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.fasterxml.jackson.datatype.joda.JodaModule]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:154)
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:102)
      	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:764)
      	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:607)
      	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:590)
      	at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.<init>(MappingJackson2HttpMessageConverter.java:57)
      	at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.<init>(AllEncompassingFormHttpMessageConverter.java:61)
      	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.<init>(RequestMappingHandlerAdapter.java:182)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
      	... 28 more
      Caused by: java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
      	at com.fasterxml.jackson.datatype.joda.ser.JodaDateSerializerBase.<clinit>(JodaDateSerializerBase.java:15)
      	at com.fasterxml.jackson.datatype.joda.JodaModule.<init>(JodaModule.java:39)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
      	... 40 more
      
      

      After doing some investigation, found the DateTimeFormatter wasn't loaded from Kylin's lib, but from:
      [Loaded org.joda.time.format.DateTimeFormatter from file:/usr/lib/hbase/lib/jruby-complete-1.6.8.jar]

      Then I remove that jar (this is a client node), restart Kylin again, this time the error disappeared, by changed to another:

      mv /usr/lib/hbase/lib/jruby-complete-1.6.8.jar /usr/lib/hbase/lib/jruby-complete-1.6.8.jar_
      
      Caused by: java.lang.ClassCastException: com.fasterxml.jackson.datatype.joda.JodaModule cannot be cast to com.fasterxml.jackson.databind.Module
      at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:764)
      at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:607)
      at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:590)
      at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.(MappingJackson2HttpMessageConverter.java:57)
      at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.(AllEncompassingFormHttpMessageConverter.java:61)
      at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.(RequestMappingHandlerAdapter.java:182)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
      ... 28 more
      
      

      Then I found it was caused by
      [Loaded com.fasterxml.jackson.datatype.joda.JodaModule from file:/usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar]

      Then rename that jar:

      mv /usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar /usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar_
      

      Restart Kylin, then it works good.

      Please avoid doing this on EMR master node, as the impact to HBase/Hive is unknown. If it is on a client node (no running service), it is okay.

       

      — Update

      Recived report that in AWS India, EMR 5.8 to start Kylin has the following error in logs/kylin.out:

       

      
      Jan 17, 2018 4:57:47 PM org.apache.catalina.core.StandardService stopInternal
      INFO: Stopping service Catalina
      Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
      INFO: Destroying Spring FrameworkServlet 'kylin'
      Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
      INFO: Closing Spring root WebApplicationContext
      Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
      INFO: Shutting down log4j
      Jan 17, 2018 4:57:47 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
      WARNING: JDBC driver de-registration failed for web application [/kylin]
      java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
      at java.sql.DriverManager.isDriverAllowed(DriverManager.java:548)
      at java.sql.DriverManager.getDrivers(DriverManager.java:446)
      at org.apache.catalina.loader.JdbcLeakPrevention.clearJdbcDriverRegistrations(JdbcLeakPrevention.java:56)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc(WebappClassLoaderBase.java:2305)
      at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:2223)
      at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:2123)
      at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:663)
      at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
      at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5875)
      at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
      at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1716)
      at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1705)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shaofengshi Shao Feng Shi
                Reporter:
                shaofengshi Shao Feng Shi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: