[KUDU-2241] NoSuchMethodError happened when run flume agent using kudu flume sink

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Component/s: flume-sink
• Assignee: Unassigned
• Reporter: Changyao Ye
• Created: 15/Dec/17 07:35
• Updated: 19/Jan/18 07:34
• Resolved: 19/Jan/18 07:34

Description

I've installed kudu and flume components from cloudera manager, and when I start flume agent the following error happened.

Error

17/12/11 10:46:29 ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error
java.lang.NoSuchMethodError: org.apache.flume.Context.getSubProperties(Ljava/lang/String;)Lorg/apache/kudu/shaded/com/google/common/collect/ImmutableMap;
at org.apache.kudu.flume.sink.KuduSink.configure(KuduSink.java:206)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:411)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:141)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Version infomation:

• flume: 1.6.0-cdh5.13.0
• kudu: 1.5.0-cdh5.13.0

I checked the classpath of flume agent and found that maybe there is version conflict of guava jar between flume and kudu flume sink(kudu-flume-sink-1.5.0-cdh5.13.0.jar) which includes shaded guava jar.

Maybe this problem related to the changes made here.

To make flume agent run properly, I excluded related class(ImmutableMap etc) from shade settings in pom.xml and found that worked.

kudu 1.5.0 top-level pom.xml

<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>org.apache.kudu.shaded.com.google.common</shadedPattern>
<excludes>
<exclude>com.google.common.collect.ImmutableMap*</exclude>
<exclude>com.google.common.collect.ImmutableEnumMap*</exclude>
</excludes>
</relocation>