Flume
  1. Flume
  2. FLUME-1735

Add support for a plugins.d directory

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      It would be nice to have a directory, called something like plugins.d, which had the following structure:

      flume/
        bin/
        conf/
        lib/
        plugins.d/
          custom-source-1/
            custom-source-1-base.jar
            custom-source-1-ext.jar
          custom-interceptor-2/
            custom-interceptor-2-main.jar
            custom-interceptor-2-lib.jar
      

      This would make it easy for users to easily install new plugins (sources, sinks, interceptors, etc) and in the case of some conflict, they can easily be differentiated and moved out of the include path. This helps with management and debugging of plugin issues, and avoids the current requirement for users to add these plugins to their FLUME_CLASSPATH.

      1. FLUME-1735-0.patch
        1 kB
        Brock Noland
      2. FLUME-1735-1.patch
        1 kB
        Brock Noland

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment -

          +1 on the general idea. I'd love to see this pluggability exposed via Bigtop so that flume users don't have to install everything but the kitchen sink.

          The bare bones implementation seems fine. A tad fancier approach would be to allow exploded jars for things that might need to be on the classpath but tweaked (sort of what wars do with WEB-INF/classes vs. WEB-INF/lib).

          Show
          Roman Shaposhnik added a comment - +1 on the general idea. I'd love to see this pluggability exposed via Bigtop so that flume users don't have to install everything but the kitchen sink. The bare bones implementation seems fine. A tad fancier approach would be to allow exploded jars for things that might need to be on the classpath but tweaked (sort of what wars do with WEB-INF/classes vs. WEB-INF/lib).
          Hide
          Brock Noland added a comment -

          Is the vision here that custom-source-1 would relate to the actual name in the configuration file? I'd think not because those should probably be logical names otherwise users would have to do a lot of file moves if they renamed their sources, sinks, etc.

          I'd think there would be an option in the cmd line, env file, or properties file which listed the directories to be added to the classpath.

          Thoughts?

          Show
          Brock Noland added a comment - Is the vision here that custom-source-1 would relate to the actual name in the configuration file? I'd think not because those should probably be logical names otherwise users would have to do a lot of file moves if they renamed their sources, sinks, etc. I'd think there would be an option in the cmd line, env file, or properties file which listed the directories to be added to the classpath. Thoughts?
          Hide
          Brock Noland added a comment -

          I am interested in this one so I will assign it to myself.

          Show
          Brock Noland added a comment - I am interested in this one so I will assign it to myself.
          Hide
          Mike Percy added a comment -

          Great! I was just thinking we could automatically add jars to the classpath and native libs to the java.library.path. I don't think we should tie it to a config naming convention. Please let me know if that makes sense to you too.

          Show
          Mike Percy added a comment - Great! I was just thinking we could automatically add jars to the classpath and native libs to the java.library.path. I don't think we should tie it to a config naming convention. Please let me know if that makes sense to you too.
          Hide
          Brock Noland added a comment -

          Hi guys,

          I thought about this as-is and it's fairly straightforward to implement. Basically in the flume-ng script for each directory we append dir/* to the classpath. This ensures those entries will come last which is likely beneficial to ensure the Flume core dependencies are not overridden.

          I would like to throw this idea out there to see what you guys thought:

          Each custom component has a child classloader. Here, in the properties file a custom component would define:

          agent.sources.custom.classpath = myplugin/*

          When it came time to load a custom component, we'd create a new classloader for that component, using the classpath defined in the configuration file. The parent classloader is the main classloader so if a component tried overriding a jar also used by flume itself, it would have no affect.

          Pros:

          • Custom Component libraries do pollute the "main" classpath.
          • Custom Components could have different, conflicting versions of dependencies.

          Cons:

          • More complex to configure.

          Thoughts?

          Show
          Brock Noland added a comment - Hi guys, I thought about this as-is and it's fairly straightforward to implement. Basically in the flume-ng script for each directory we append dir/* to the classpath. This ensures those entries will come last which is likely beneficial to ensure the Flume core dependencies are not overridden. I would like to throw this idea out there to see what you guys thought: Each custom component has a child classloader. Here, in the properties file a custom component would define: agent.sources.custom.classpath = myplugin/* When it came time to load a custom component, we'd create a new classloader for that component, using the classpath defined in the configuration file. The parent classloader is the main classloader so if a component tried overriding a jar also used by flume itself, it would have no affect. Pros: Custom Component libraries do pollute the "main" classpath. Custom Components could have different, conflicting versions of dependencies. Cons: More complex to configure. Thoughts?
          Hide
          Mike Percy added a comment -

          It what you are describing something like OSGi? The direction sounds like the right way to go, to me, in the long term. In the short term, just starting off with something that, as you say, adds certain subdirectories to the classpath at startup time seems like a win.

          Show
          Mike Percy added a comment - It what you are describing something like OSGi? The direction sounds like the right way to go, to me, in the long term. In the short term, just starting off with something that, as you say, adds certain subdirectories to the classpath at startup time seems like a win.
          Hide
          Brock Noland added a comment -

          Hi,

          I agree that OSGi is the correct mechanism in the long term. This wouldn't be anything nearly as expansive as OSGi. This proposal would simply create a separate classloader for the custom components. I think I should be able to whip up a patch in an hour or so. I'll try that and if I hit a roadblock or it takes too long I'll switch to the original proposal.

          Brock

          Show
          Brock Noland added a comment - Hi, I agree that OSGi is the correct mechanism in the long term. This wouldn't be anything nearly as expansive as OSGi. This proposal would simply create a separate classloader for the custom components. I think I should be able to whip up a patch in an hour or so. I'll try that and if I hit a roadblock or it takes too long I'll switch to the original proposal. Brock
          Hide
          Brock Noland added a comment -

          Hi guys,

          Attached is a POC of what I am referring to. It's only implemented for sources. The TestClassloaders class should be interesting.

          Additionally, I have it working here: http://people.apache.org/~brock/FLUME-1735-POC-0.tar.gz

          Basically the tar.gz has flume built with the change, a custom jar, and configuration file. In the conf file I have added the classpath which is used to load the classes (a few examples):

          agent.sources.src-1.type = org.apache.flume.source.CustomStressSource
          agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/lib/*
          #agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/classes
          #agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/lib/custom.jar
          
          Show
          Brock Noland added a comment - Hi guys, Attached is a POC of what I am referring to. It's only implemented for sources. The TestClassloaders class should be interesting. Additionally, I have it working here: http://people.apache.org/~brock/FLUME-1735-POC-0.tar.gz Basically the tar.gz has flume built with the change, a custom jar, and configuration file. In the conf file I have added the classpath which is used to load the classes (a few examples): agent.sources.src-1.type = org.apache.flume.source.CustomStressSource agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/lib/* #agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/classes #agent.sources.src-1.classpath = /Users/noland/tmp/FLUME-1735/lib/custom.jar
          Hide
          Brock Noland added a comment -

          Hi guys,

          So after thinking about this, I think it's still a good idea, but I also think we should take some time and have some serious design discussions around it. As such, I think we should implement the plugins.d functionality and then revisit the classloader stuff in another JIRA.

          Brock

          Show
          Brock Noland added a comment - Hi guys, So after thinking about this, I think it's still a good idea, but I also think we should take some time and have some serious design discussions around it. As such, I think we should implement the plugins.d functionality and then revisit the classloader stuff in another JIRA. Brock
          Hide
          Roman Shaposhnik added a comment -

          +1 on doing some additional design/planning around classloader approach. As for the plugins.d – is there any chance to make it fully dynamic? As in – avoid any need for
          extra setting in the configuration file?

          Show
          Roman Shaposhnik added a comment - +1 on doing some additional design/planning around classloader approach. As for the plugins.d – is there any chance to make it fully dynamic? As in – avoid any need for extra setting in the configuration file?
          Hide
          Brock Noland added a comment -

          Hi,

          Let me know what you think about the attached patch. It looks for a plugin.d directory in FLUME_HOME. Each plugin (subdirectory) could have up to three sub directories, lib (plugin itself), libext (libs for the plugin), and native (.so's).

          $ find plugins.d
          plugins.d
          plugins.d/custom-source-0
          plugins.d/custom-source-0/lib
          plugins.d/custom-source-0/libext
          plugins.d/custom-source-1
          plugins.d/custom-source-1/lib
          plugins.d/custom-source-1/lib/custom.jar
          plugins.d/custom-source-1/native
          

          which results in the following command:

          java -Xmx20m -cp '/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-0/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-1/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-0/libext/*:/opt/local/hadoop.... 
          -Djava.library.path=/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-1/native:/opt/local/hadoop...
          

          If that looks good I can add documentation to the patch.

          Show
          Brock Noland added a comment - Hi, Let me know what you think about the attached patch. It looks for a plugin.d directory in FLUME_HOME. Each plugin (subdirectory) could have up to three sub directories, lib (plugin itself), libext (libs for the plugin), and native (.so's). $ find plugins.d plugins.d plugins.d/custom-source-0 plugins.d/custom-source-0/lib plugins.d/custom-source-0/libext plugins.d/custom-source-1 plugins.d/custom-source-1/lib plugins.d/custom-source-1/lib/custom.jar plugins.d/custom-source-1/native which results in the following command: java -Xmx20m -cp '/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-0/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-1/lib/*:/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-0/libext/*:/opt/local/hadoop.... -Djava.library.path=/Users/noland/tmp/FLUME-1735-POC-0/apache-flume-1.4.0-SNAPSHOT-bin/plugins.d/custom-source-1/native:/opt/local/hadoop... If that looks good I can add documentation to the patch.
          Hide
          Brock Noland added a comment -

          Ha, we commented during the same minute! Yes, the patch I posted is fully dynamic.

          Show
          Brock Noland added a comment - Ha, we commented during the same minute! Yes, the patch I posted is fully dynamic.
          Hide
          Mike Percy added a comment -

          Looks good! One thing, let's set plugin_lib="", plugin_libext="", plugin_native="" at the top of the file so we don't inherit those from the shell.

          Show
          Mike Percy added a comment - Looks good! One thing, let's set plugin_lib="", plugin_libext="", plugin_native="" at the top of the file so we don't inherit those from the shell.
          Hide
          Brock Noland added a comment -

          I'll add

          unset plugin_lib plugin_libext plugin_native

          Show
          Brock Noland added a comment - I'll add unset plugin_lib plugin_libext plugin_native
          Hide
          Brock Noland added a comment -

          Updated patch

          Show
          Brock Noland added a comment - Updated patch
          Hide
          Mike Percy added a comment -

          +1 lgtm

          Show
          Mike Percy added a comment - +1 lgtm
          Hide
          Mike Percy added a comment -

          Pushed to trunk & flume-1.4 branch. Thanks for the patch, Brock!

          Trunk rev: 8522cfef78fe72d73f5127bd3fcdf00e7a9d383d

          Show
          Mike Percy added a comment - Pushed to trunk & flume-1.4 branch. Thanks for the patch, Brock! Trunk rev: 8522cfef78fe72d73f5127bd3fcdf00e7a9d383d
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/)
          FLUME-1735. Add support for a plugins.d directory. (Revision 8522cfef78fe72d73f5127bd3fcdf00e7a9d383d)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=8522cfef78fe72d73f5127bd3fcdf00e7a9d383d
          Files :

          • bin/flume-ng
          Show
          Hudson added a comment - Integrated in flume-trunk #344 (See https://builds.apache.org/job/flume-trunk/344/ ) FLUME-1735 . Add support for a plugins.d directory. (Revision 8522cfef78fe72d73f5127bd3fcdf00e7a9d383d) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=8522cfef78fe72d73f5127bd3fcdf00e7a9d383d Files : bin/flume-ng

            People

            • Assignee:
              Brock Noland
              Reporter:
              Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development