[KYLIN-3129] Fix the joda library conflicts during Kylin start on EMR 5.8+

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Workaround
• Fix Version/s: v2.3.0
• Component/s: Environment
• Assignee: Shao Feng Shi
• Reporter: Shao Feng Shi
• Created: 22/Dec/17 06:54
• Updated: 02/Apr/18 13:42
• Resolved: 22/Dec/17 09:45

Description

When run "bin/kylin.sh start", it reports an NoSuchMethodError:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.fasterxml.jackson.datatype.joda.JodaModule]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:154)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:102)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:764)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:607)
	at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:590)
	at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.<init>(MappingJackson2HttpMessageConverter.java:57)
	at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.<init>(AllEncompassingFormHttpMessageConverter.java:61)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.<init>(RequestMappingHandlerAdapter.java:182)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	... 28 more
Caused by: java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
	at com.fasterxml.jackson.datatype.joda.ser.JodaDateSerializerBase.<clinit>(JodaDateSerializerBase.java:15)
	at com.fasterxml.jackson.datatype.joda.JodaModule.<init>(JodaModule.java:39)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
	... 40 more

After doing some investigation, found the DateTimeFormatter wasn't loaded from Kylin's lib, but from:
[Loaded org.joda.time.format.DateTimeFormatter from file:/usr/lib/hbase/lib/jruby-complete-1.6.8.jar]

Then I remove that jar (this is a client node), restart Kylin again, this time the error disappeared, by changed to another:

mv /usr/lib/hbase/lib/jruby-complete-1.6.8.jar /usr/lib/hbase/lib/jruby-complete-1.6.8.jar_

Caused by: java.lang.ClassCastException: com.fasterxml.jackson.datatype.joda.JodaModule cannot be cast to com.fasterxml.jackson.databind.Module
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable(Jackson2ObjectMapperBuilder.java:764)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.configure(Jackson2ObjectMapperBuilder.java:607)
at org.springframework.http.converter.json.Jackson2ObjectMapperBuilder.build(Jackson2ObjectMapperBuilder.java:590)
at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.(MappingJackson2HttpMessageConverter.java:57)
at org.springframework.http.converter.support.AllEncompassingFormHttpMessageConverter.(AllEncompassingFormHttpMessageConverter.java:61)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.(RequestMappingHandlerAdapter.java:182)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
... 28 more

Then I found it was caused by
[Loaded com.fasterxml.jackson.datatype.joda.JodaModule from file:/usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar]

Then rename that jar:

mv /usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar /usr/lib/hive/lib/jackson-datatype-joda-2.4.6.jar_

Restart Kylin, then it works good.

Please avoid doing this on EMR master node, as the impact to HBase/Hive is unknown. If it is on a client node (no running service), it is okay.

 

— Update

Recived report that in AWS India, EMR 5.8 to start Kylin has the following error in logs/kylin.out:

 

Jan 17, 2018 4:57:47 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
INFO: Destroying Spring FrameworkServlet 'kylin'
Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jan 17, 2018 4:57:47 PM org.apache.catalina.core.ApplicationContext log
INFO: Shutting down log4j
Jan 17, 2018 4:57:47 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
WARNING: JDBC driver de-registration failed for web application [/kylin]
java.lang.NoClassDefFoundError: Could not initialize class org.apache.derby.jdbc.AutoloadedDriver40
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.sql.DriverManager.isDriverAllowed(DriverManager.java:556)
at java.sql.DriverManager.isDriverAllowed(DriverManager.java:548)
at java.sql.DriverManager.getDrivers(DriverManager.java:446)
at org.apache.catalina.loader.JdbcLeakPrevention.clearJdbcDriverRegistrations(JdbcLeakPrevention.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc(WebappClassLoaderBase.java:2305)
at org.apache.catalina.loader.WebappClassLoaderBase.clearReferences(WebappClassLoaderBase.java:2223)
at org.apache.catalina.loader.WebappClassLoaderBase.stop(WebappClassLoaderBase.java:2123)
at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:663)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5875)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1716)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1705)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Issue Links

• is duplicated by: KYLIN-3176 NoClassDefFoundError for Derby library on EMR 5.8