Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1677

Add File-channel dependency to flume-ng-node's pom.xml

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.4.0
    • Component/s: Build
    • Labels:
      None

      Description

      flume-ng-node/pom.xml is missing dependency on file channel. Leads to class not found 'org.apache.flume.channel.file.FileChannel' exception when running/debugging Flume within IDEs (if file channel is being used).

      1. FLUME-1677.patch
        0.4 kB
        Roshan Naik

        Activity

        Hide
        roshan_naik Roshan Naik added a comment -

        Added file-channel dependency in flume-ng-node/pom.xml

        Show
        roshan_naik Roshan Naik added a comment - Added file-channel dependency in flume-ng-node/pom.xml
        Hide
        roshan_naik Roshan Naik added a comment -

        file-channel dependency was missing in flume-ng-node/pom.xml

        Show
        roshan_naik Roshan Naik added a comment - file-channel dependency was missing in flume-ng-node/pom.xml
        Hide
        roshan_naik Roshan Naik added a comment -

        This one is a trivial one to review.. can somebody take a look. thanks. i run into issues with intellij without this.

        Show
        roshan_naik Roshan Naik added a comment - This one is a trivial one to review.. can somebody take a look. thanks. i run into issues with intellij without this.
        Hide
        mpercy Mike Percy added a comment - - edited

        Can you clarify exactly where file channel is being referenced in the node module? I don't see it in there:

        {mpercy@silverdoom}$ pwd
        /Users/mpercy/src/flume/flume-ng-node/src
        {mpercy@silverdoom}$ find . -type f -name FileChannel
        {mpercy@silverdoom}$ 
        

        Maybe it's in a transitive dependency, which should be fixed in this fashion instead of declaring an explicit dependency on a transitive dep.

        Show
        mpercy Mike Percy added a comment - - edited Can you clarify exactly where file channel is being referenced in the node module? I don't see it in there: {mpercy@silverdoom}$ pwd /Users/mpercy/src/flume/flume-ng-node/src {mpercy@silverdoom}$ find . -type f -name FileChannel {mpercy@silverdoom}$ Maybe it's in a transitive dependency, which should be fixed in this fashion instead of declaring an explicit dependency on a transitive dep.
        Hide
        mpercy Mike Percy added a comment - - edited

        Um, that wouldn't find what I was looking for. This would though. And it still comes up empty...

        {mpercy@silverdoom}$ grep -r FileChannel .
        {mpercy@silverdoom}$ 
        
        Show
        mpercy Mike Percy added a comment - - edited Um, that wouldn't find what I was looking for. This would though. And it still comes up empty... {mpercy@silverdoom}$ grep -r FileChannel . {mpercy@silverdoom}$
        Hide
        roshan_naik Roshan Naik added a comment -

        Mike,
        Not sure if I understand the question. The point here is that the File channel reference is missing ... naturally you wont find it. A 'class not found' problem is enconutered when trying to step through the file channel in a debugger from the sources location (but not an issue in the deployed version).

        Take a look at flume-ng-node/pom.xml. It contains has a list of other channels in the dependencies. File channel is missing there.

        Show
        roshan_naik Roshan Naik added a comment - Mike, Not sure if I understand the question. The point here is that the File channel reference is missing ... naturally you wont find it. A 'class not found' problem is enconutered when trying to step through the file channel in a debugger from the sources location (but not an issue in the deployed version). Take a look at flume-ng-node/pom.xml. It contains has a list of other channels in the dependencies. File channel is missing there.
        Hide
        mpercy Mike Percy added a comment -

        If you wouldn't mind, please post the full stack trace. My last comment shows a grep and FileChannel is nowhere to be found there. We should only declare dependencies where they are directly referenced in the source code. I don't doubt some dep is missing from somewhere. Please post the entire stack trace, and lets fix the appropriate dep in the right place.

        Show
        mpercy Mike Percy added a comment - If you wouldn't mind, please post the full stack trace. My last comment shows a grep and FileChannel is nowhere to be found there. We should only declare dependencies where they are directly referenced in the source code. I don't doubt some dep is missing from somewhere. Please post the entire stack trace, and lets fix the appropriate dep in the right place.
        Hide
        roshan_naik Roshan Naik added a comment -

        I captured this stack trace by introducing a try/catch inside PropertiesFileConfigurationProvider.loadChannels()
        This exception normally gets swallowed.

        org.apache.flume.FlumeException: Unable to load channel type: org.apache.flume.channel.file.FileChannel, class: org.apache.flume.channel.file.FileChannel
        	at org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:87)
        	at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:269)
        	at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221)
        	at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
        	at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
        	at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        	at java.lang.Thread.run(Thread.java:680)
        Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
        	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        	at java.lang.Class.forName0(Native Method)
        	at java.lang.Class.forName(Class.java:169)
        	at org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:85)
        	... 14 more
        
        
        Show
        roshan_naik Roshan Naik added a comment - I captured this stack trace by introducing a try/catch inside PropertiesFileConfigurationProvider.loadChannels() This exception normally gets swallowed. org.apache.flume.FlumeException: Unable to load channel type: org.apache.flume.channel.file.FileChannel, class: org.apache.flume.channel.file.FileChannel at org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:87) at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:269) at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:221) at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123) at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38) at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang. Thread .run( Thread .java:680) Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang. ClassLoader .loadClass( ClassLoader .java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang. ClassLoader .loadClass( ClassLoader .java:247) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:169) at org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:85) ... 14 more
        Hide
        roshan_naik Roshan Naik added a comment -

        its unclear to me why dynamically loaded stuff (like channels) should not be referenced in pom.xml as a dependencies.

        Show
        roshan_naik Roshan Naik added a comment - its unclear to me why dynamically loaded stuff (like channels) should not be referenced in pom.xml as a dependencies.
        Hide
        mpercy Mike Percy added a comment -

        +1. I'm a little conflicted, maybe this is something that should be configured @ the IDE level, but it shouldn't hurt anything.

        Show
        mpercy Mike Percy added a comment - +1. I'm a little conflicted, maybe this is something that should be configured @ the IDE level, but it shouldn't hurt anything.
        Hide
        mpercy Mike Percy added a comment -

        Pushed to trunk and flume-1.4 branches. Thanks Roshan!

        Show
        mpercy Mike Percy added a comment - Pushed to trunk and flume-1.4 branches. Thanks Roshan!
        Hide
        hudson Hudson added a comment -

        Integrated in flume-trunk #444 (See https://builds.apache.org/job/flume-trunk/444/)
        FLUME-1677. Add File-channel dependency to flume-ng-node's pom.xml. (Revision 83e4fe5f7a15ac0c4c8b5aee5e910169e189b051)

        Result = FAILURE
        mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=83e4fe5f7a15ac0c4c8b5aee5e910169e189b051
        Files :

        • flume-ng-node/pom.xml
        Show
        hudson Hudson added a comment - Integrated in flume-trunk #444 (See https://builds.apache.org/job/flume-trunk/444/ ) FLUME-1677 . Add File-channel dependency to flume-ng-node's pom.xml. (Revision 83e4fe5f7a15ac0c4c8b5aee5e910169e189b051) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=83e4fe5f7a15ac0c4c8b5aee5e910169e189b051 Files : flume-ng-node/pom.xml

          People

          • Assignee:
            roshan_naik Roshan Naik
            Reporter:
            roshan_naik Roshan Naik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development