Flume
  1. Flume
  2. FLUME-2377

Don't automatically include Hadoop and HBase classpaths into Flume classpath

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Shell
    • Labels:
      None

      Description

      HBase and Hadoop, for their own reasons, have Jersey 1.x libraries in their classpath. A custom source we wrote uses Jersey 2.x. For a number of reasons, the two versions of Jersey do not co-habitate on the same classpath nicely. As a result, we get stacktraces like the following if, for example, a developer has Hadoop or HBase installed on their system:

      java.lang.NullPointerException
          at com.sun.jersey.core.provider.jaxb.AbstractJAXBProvider.setConfiguration(AbstractJAXBProvider.java:107)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
          at org.jvnet.hk2.internal.ClazzCreator.methodMe(ClazzCreator.java:374)
          at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:427)
          at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
          at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:69)
          at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
          at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
          at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
          at org.glassfish.jersey.internal.inject.ProviderToService.apply(ProviderToService.java:58)
          at org.glassfish.jersey.internal.inject.ProviderToService.apply(ProviderToService.java:54)
          at jersey.repackaged.com.google.common.collect.Iterators$8.transform(Iterators.java:860)
          at jersey.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
          at java.util.AbstractCollection.addAll(AbstractCollection.java:333)
          at java.util.LinkedHashSet.<init>(LinkedHashSet.java:169)
          at jersey.repackaged.com.google.common.collect.Sets.newLinkedHashSet(Sets.java:292)
          at org.glassfish.jersey.internal.inject.Providers.getClasses(Providers.java:364)
          at org.glassfish.jersey.internal.inject.Providers.getProviders(Providers.java:186)
          at org.glassfish.jersey.message.internal.MessageBodyFactory.<init>(MessageBodyFactory.java:304)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
          at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1105)
          at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:291)
          at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:421)
          at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
          at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:114)
          at org.jvnet.hk2.internal.SingletonContext$1.compute(SingletonContext.java:102)
          at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture$1.call(Cache.java:97)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
          at org.glassfish.hk2.utilities.cache.Cache$OriginThreadAwareFuture.run(Cache.java:154)
          at org.glassfish.hk2.utilities.cache.Cache.compute(Cache.java:199)
          at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:153)
          at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:629)
          at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:614)
          at org.glassfish.jersey.server.model.ComponentModelValidator.<init>(ComponentModelValidator.java:97)
          at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:457)
          at org.glassfish.jersey.server.ApplicationHandler.access$500(ApplicationHandler.java:162)
          at org.glassfish.jersey.server.ApplicationHandler$3.run(ApplicationHandler.java:322)
          at org.glassfish.jersey.internal.Errors$2.call(Errors.java:289)
          at org.glassfish.jersey.internal.Errors$2.call(Errors.java:286)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
          at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
          at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:286)
          at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:319)
          at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:272)
          at org.glassfish.jersey.server.ContainerFactory.createContainer(ContainerFactory.java:79)
          at org.glassfish.jersey.jetty.JettyHttpContainerFactory.createServer(JettyHttpContainerFactory.java:139)
          at com.me.CustomSource.start(CustomSource.java:186)
          at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
          at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:724)
      

      The fact that flume-ng includes the Hadoop and HBase classpath automatically, even if we have no need for it, is a bit surprising and, if you don't know that the shell script is doing that, it can take a little bit of digging to figure it out.

      At the very least, I'd like the ability to tell flume-ng, "No, don't include the Hadoop classpath," and, "No, don't include the HBase classpath,", though if someone's willing to throw passivity to the wind, I'd be glad to just not have those external classpaths added to Flume's classpath to begin with.

        Activity

        Joshua Hyde created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Joshua Hyde
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development