Flume
  1. Flume
  2. FLUME-932

Making flume-ng components pluggage and name aware

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.0.0
    • Fix Version/s: v1.1.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently the components in flume-ng are hard wired to use the built in name. This implies that within one agent, there can be only one instance of a particular component. While the configuration supports having multiple components of the same time, the component interfaces themselves need to change.

      Also, need to support plugin components for user-provided implementations of source/sink/channels where necessary.

      1. FLUME-932-2-svn.patch
        87 kB
        Arvind Prabhakar
      2. FLUME-932-1-svn.patch
        84 kB
        Arvind Prabhakar

        Activity

        Arvind Prabhakar created issue -
        Arvind Prabhakar made changes -
        Field Original Value New Value
        Attachment FLUME-932-1-svn.patch [ 12510472 ]
        Arvind Prabhakar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3482/
        -----------------------------------------------------------

        Review request for Flume and Prasad Mujumdar.

        Summary
        -------

        Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism (FLUME-930).

        This addresses bug FLUME-932.
        https://issues.apache.org/jira/browse/FLUME-932

        Diffs


        flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6
        flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75
        flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7
        flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7
        flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95
        flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f
        flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17
        flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3
        flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785
        flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c
        flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab
        flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7
        flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9
        flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb
        flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655
        flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf
        flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8
        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53
        flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9
        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959
        flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7
        flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0
        flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31
        flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1
        flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4
        flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086
        flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809
        flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0
        flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b
        flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f
        flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f
        flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a
        flume-ng-node/src/test/resources/flume-conf.json 0449142

        Diff: https://reviews.apache.org/r/3482/diff

        Testing
        -------

        All tests ran successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/ ----------------------------------------------------------- Review request for Flume and Prasad Mujumdar. Summary ------- Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism ( FLUME-930 ). This addresses bug FLUME-932 . https://issues.apache.org/jira/browse/FLUME-932 Diffs flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6 flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75 flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7 flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7 flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95 flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17 flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3 flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785 flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7 flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9 flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655 flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53 flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9 flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959 flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7 flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0 flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31 flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1 flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4 flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809 flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0 flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a flume-ng-node/src/test/resources/flume-conf.json 0449142 Diff: https://reviews.apache.org/r/3482/diff Testing ------- All tests ran successfully. Thanks, Arvind
        Arvind Prabhakar made changes -
        Attachment FLUME-932-2-svn.patch [ 12510526 ]
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3482/
        -----------------------------------------------------------

        (Updated 2012-01-13 20:40:40.257009)

        Review request for Flume and Prasad Mujumdar.

        Changes
        -------

        Did some manual testing of failure scenario, found a few bugs with the change and fixed them. I was able to simulate a backlogged channel and drain it at a later stage without any problems.

        Summary
        -------

        Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism (FLUME-930).

        This addresses bug FLUME-932.
        https://issues.apache.org/jira/browse/FLUME-932

        Diffs (updated)


        flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a
        flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75
        flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7
        flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7
        flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95
        flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f
        flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17
        flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3
        flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785
        flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c
        flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab
        flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7
        flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9
        flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION
        flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb
        flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655
        flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf
        flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8
        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53
        flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9
        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959
        flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION
        flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7
        flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0
        flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31
        flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1
        flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4
        flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086
        flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809
        flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0
        flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b
        flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f
        flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f
        flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a
        flume-ng-node/src/test/resources/flume-conf.json 0449142

        Diff: https://reviews.apache.org/r/3482/diff

        Testing
        -------

        All tests ran successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/ ----------------------------------------------------------- (Updated 2012-01-13 20:40:40.257009) Review request for Flume and Prasad Mujumdar. Changes ------- Did some manual testing of failure scenario, found a few bugs with the change and fixed them. I was able to simulate a backlogged channel and drain it at a later stage without any problems. Summary ------- Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism ( FLUME-930 ). This addresses bug FLUME-932 . https://issues.apache.org/jira/browse/FLUME-932 Diffs (updated) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75 flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7 flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7 flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95 flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17 flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3 flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785 flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7 flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9 flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655 flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53 flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9 flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959 flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7 flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0 flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31 flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1 flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4 flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809 flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0 flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a flume-ng-node/src/test/resources/flume-conf.json 0449142 Diff: https://reviews.apache.org/r/3482/diff Testing ------- All tests ran successfully. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3482/#review4373
        -----------------------------------------------------------

        Ship it!

        Looks good overall.
        Minor comment -
        The factories allow you to call create() multiple times for a same source name/type and return the source if its already instantiated, on the other hand the unregister will remove it on first invocation. We should perhaps use refcounting or raise error on in create if object already exists.

        • Prasad

        On 2012-01-13 20:40:40, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3482/

        -----------------------------------------------------------

        (Updated 2012-01-13 20:40:40)

        Review request for Flume and Prasad Mujumdar.

        Summary

        -------

        Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism (FLUME-930).

        This addresses bug FLUME-932.

        https://issues.apache.org/jira/browse/FLUME-932

        Diffs

        -----

        flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a

        flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75

        flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7

        flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7

        flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95

        flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f

        flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17

        flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3

        flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785

        flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c

        flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab

        flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7

        flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9

        flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb

        flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655

        flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf

        flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8

        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53

        flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959

        flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7

        flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0

        flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31

        flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1

        flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4

        flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086

        flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809

        flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0

        flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b

        flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f

        flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f

        flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a

        flume-ng-node/src/test/resources/flume-conf.json 0449142

        Diff: https://reviews.apache.org/r/3482/diff

        Testing

        -------

        All tests ran successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/#review4373 ----------------------------------------------------------- Ship it! Looks good overall. Minor comment - The factories allow you to call create() multiple times for a same source name/type and return the source if its already instantiated, on the other hand the unregister will remove it on first invocation. We should perhaps use refcounting or raise error on in create if object already exists. Prasad On 2012-01-13 20:40:40, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/ ----------------------------------------------------------- (Updated 2012-01-13 20:40:40) Review request for Flume and Prasad Mujumdar. Summary ------- Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism ( FLUME-930 ). This addresses bug FLUME-932 . https://issues.apache.org/jira/browse/FLUME-932 Diffs ----- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75 flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7 flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7 flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95 flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17 flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3 flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785 flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7 flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9 flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655 flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53 flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9 flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959 flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7 flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0 flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31 flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1 flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4 flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809 flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0 flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a flume-ng-node/src/test/resources/flume-conf.json 0449142 Diff: https://reviews.apache.org/r/3482/diff Testing ------- All tests ran successfully. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-01-13 22:52:22, Prasad Mujumdar wrote:

        > Looks good overall.

        > Minor comment -

        > The factories allow you to call create() multiple times for a same source name/type and return the source if its already instantiated, on the other hand the unregister will remove it on first invocation. We should perhaps use refcounting or raise error on in create if object already exists.

        >

        Thanks Prasad. I have filed https://issues.apache.org/jira/browse/FLUME-933 to track this change.

        • Arvind

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/3482/#review4373
        -----------------------------------------------------------

        On 2012-01-13 20:40:40, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/3482/

        -----------------------------------------------------------

        (Updated 2012-01-13 20:40:40)

        Review request for Flume and Prasad Mujumdar.

        Summary

        -------

        Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism (FLUME-930).

        This addresses bug FLUME-932.

        https://issues.apache.org/jira/browse/FLUME-932

        Diffs

        -----

        flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a

        flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75

        flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7

        flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7

        flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95

        flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f

        flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17

        flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3

        flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785

        flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c

        flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab

        flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7

        flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9

        flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION

        flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb

        flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655

        flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf

        flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8

        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53

        flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959

        flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION

        flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7

        flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0

        flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31

        flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1

        flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4

        flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086

        flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809

        flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0

        flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b

        flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f

        flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f

        flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a

        flume-ng-node/src/test/resources/flume-conf.json 0449142

        Diff: https://reviews.apache.org/r/3482/diff

        Testing

        -------

        All tests ran successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-01-13 22:52:22, Prasad Mujumdar wrote: > Looks good overall. > Minor comment - > The factories allow you to call create() multiple times for a same source name/type and return the source if its already instantiated, on the other hand the unregister will remove it on first invocation. We should perhaps use refcounting or raise error on in create if object already exists. > Thanks Prasad. I have filed https://issues.apache.org/jira/browse/FLUME-933 to track this change. Arvind ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/#review4373 ----------------------------------------------------------- On 2012-01-13 20:40:40, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3482/ ----------------------------------------------------------- (Updated 2012-01-13 20:40:40) Review request for Flume and Prasad Mujumdar. Summary ------- Modified all the factories and their default implementations to use specific Enums (newly added) to locate the plugin classes. See SourceType, SinkType, ChannelType for details. Modified the Source interface to directly allow a list of channels. This decommissions the fan-out channel that will be eventually replaced by the channel selector mechanism ( FLUME-930 ). This addresses bug FLUME-932 . https://issues.apache.org/jira/browse/FLUME-932 Diffs ----- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java 6868ae4 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java cdaf6d6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 10916c6 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java f111392 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java eab655a flume-ng-core/src/main/java/org/apache/flume/Channel.java 6757b75 flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java 32233b7 flume-ng-core/src/main/java/org/apache/flume/FlumeException.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/Sink.java 49ba7c7 flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java 22b1e95 flume-ng-core/src/main/java/org/apache/flume/Source.java 410529f flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java ce51c17 flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 6de7bb3 flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java 7832785 flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java 7305b6c flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 0635aab flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java f6366d7 flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java 88739e9 flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java 3e8d5fb flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java f20d655 flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java 01f57cf flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java 3d5d7d8 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 5275f53 flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java e4933e9 flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java fd09959 flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java PRE-CREATION flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java ec6bbb7 flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java 2bfb9a0 flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 859dd31 flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java f1522d1 flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 4383db4 flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java e146086 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 12ea809 flume-ng-node/src/main/java/org/apache/flume/node/Application.java 54ab4e0 flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java 498277b flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 464a32f flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java cc5814f flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java 4af755a flume-ng-node/src/test/resources/flume-conf.json 0449142 Diff: https://reviews.apache.org/r/3482/diff Testing ------- All tests ran successfully. Thanks, Arvind
        Hide
        Hudson added a comment -

        Integrated in flume-728 #94 (See https://builds.apache.org/job/flume-728/94/)
        FLUME-932. Making Flume-NG components name-aware and pluggable.

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1231371
        Files :

        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Channel.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/FlumeException.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Sink.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Source.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/node/Application.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
        • /incubator/flume/branches/flume-728/flume-ng-node/src/test/resources/flume-conf.json
        Show
        Hudson added a comment - Integrated in flume-728 #94 (See https://builds.apache.org/job/flume-728/94/ ) FLUME-932 . Making Flume-NG components name-aware and pluggable. arvind : http://svn.apache.org/viewvc/?view=rev&rev=1231371 Files : /incubator/flume/branches/flume-728/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcTransactionImpl.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/PersistableEvent.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Channel.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/ChannelFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/FlumeException.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/NamedComponent.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Sink.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/SinkFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Source.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/SourceFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/FanoutChannel.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AbstractSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/sink/TestDefaultSinkFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/MockSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java /incubator/flume/branches/flume-728/flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/file/JsonFileConfigurationProvider.java /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java /incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/node/Application.java /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/conf/file/TestJsonFileConfigurationProvider.java /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/node/TestDefaultLogicalNodeManager.java /incubator/flume/branches/flume-728/flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java /incubator/flume/branches/flume-728/flume-ng-node/src/test/resources/flume-conf.json
        Hide
        E. Sammer added a comment -

        I know this was committed already and I missed it, but I'm not so sure are ideal here.

        This seems to deal with the case that one can't have two plugin definitions for the same name. I'm not sure how this matters though. For instance, the name "memory" only allows one to create new instances of MemoryChannel that are unrelated; there's nothing else shared here.

        The other issue is that we now use enums to create instances of various components. This means that those creating new components are going to be tempted to add items to the enum. This is pretty gross. I have a really, really strong preference for reverting this. If this is because there's a desire to have spring-like functionality, let's just use spring.

        Show
        E. Sammer added a comment - I know this was committed already and I missed it, but I'm not so sure are ideal here. This seems to deal with the case that one can't have two plugin definitions for the same name. I'm not sure how this matters though. For instance, the name "memory" only allows one to create new instances of MemoryChannel that are unrelated; there's nothing else shared here. The other issue is that we now use enums to create instances of various components. This means that those creating new components are going to be tempted to add items to the enum. This is pretty gross. I have a really, really strong preference for reverting this. If this is because there's a desire to have spring-like functionality, let's just use spring.
        Hide
        Arvind Prabhakar added a comment -

        Eric, this primary objective of this change was to allow name support for components. So an avro source can be named "myavrosrc" instead of being required to call it "avro" only. It also enables us to have multiple avro sources that are named differently within the same agent (myavrosrc1, myavrosrc2) and be able to configure them differently. It is not an intent, but a side-effect, that you cannot have two plugins of the same type that are named the same.

        The other issue is that we now use enums to create instances of various components. This means that those creating new components are going to be tempted to add items to the enum.

        The enum here is a convenient idiom to have one place registration of short-names to class-name mapping for various components. What was previously being done in Application.loadPlugins() method is now externalized into these enums. These enums are only there to enumerate the built-in components and do not in anyway inhibit the use of drop-in components. The convenience they offer is to allow the user to specify the type of a source as avro, instead of its fully qualified class name.

        Show
        Arvind Prabhakar added a comment - Eric, this primary objective of this change was to allow name support for components. So an avro source can be named "myavrosrc" instead of being required to call it "avro" only. It also enables us to have multiple avro sources that are named differently within the same agent (myavrosrc1, myavrosrc2) and be able to configure them differently. It is not an intent, but a side-effect, that you cannot have two plugins of the same type that are named the same. The other issue is that we now use enums to create instances of various components. This means that those creating new components are going to be tempted to add items to the enum. The enum here is a convenient idiom to have one place registration of short-names to class-name mapping for various components. What was previously being done in Application.loadPlugins() method is now externalized into these enums. These enums are only there to enumerate the built-in components and do not in anyway inhibit the use of drop-in components. The convenience they offer is to allow the user to specify the type of a source as avro, instead of its fully qualified class name.
        Hide
        Ralph Goers added a comment -

        Pardon me for also jumping in late but I also have a number of issues with this. Mostly they revolve around the thread safety of the code.

        1. All the methods in DefaultSourceFactory are synchronized. All the methods in DefaultChannelFactory are not. DefaultChannelFactory isn't using a ConcurrentHashMap so it doesn't appear to be thread safe. While DefaultSourceFactory synchronizes all of its methods this could probably be avoided if a ConcurrentHashMap was used.
        2. Methods such as setName and getName on AbstractChannel are synchronized. This is pointless since the operation they are performing is atomic.
        3. It would be much better if setter methods were avoided entirely and the values were required to be passed to the constructor instead. This would make the objects largely immutable and much easier to deal with in a thread-safe manner. For example, just because the setName method is synchronized doesn't prevent the method from being called after the object has already been added to the configuration. Preventing that in the set method itself requires all kinds of weird checks to cause an IllegalStateException to be thrown.
        4. Although it may not have been modified here, I'm also confused by DefaultLogicalNodeManager's onNodeConfigurationChanged method. I see it calling the LifecycleSupervisor for each of the Sinks and Sources in the new configuration. What does it do with the Sinks and Sources from the previous configuration?

        Show
        Ralph Goers added a comment - Pardon me for also jumping in late but I also have a number of issues with this. Mostly they revolve around the thread safety of the code. 1. All the methods in DefaultSourceFactory are synchronized. All the methods in DefaultChannelFactory are not. DefaultChannelFactory isn't using a ConcurrentHashMap so it doesn't appear to be thread safe. While DefaultSourceFactory synchronizes all of its methods this could probably be avoided if a ConcurrentHashMap was used. 2. Methods such as setName and getName on AbstractChannel are synchronized. This is pointless since the operation they are performing is atomic. 3. It would be much better if setter methods were avoided entirely and the values were required to be passed to the constructor instead. This would make the objects largely immutable and much easier to deal with in a thread-safe manner. For example, just because the setName method is synchronized doesn't prevent the method from being called after the object has already been added to the configuration. Preventing that in the set method itself requires all kinds of weird checks to cause an IllegalStateException to be thrown. 4. Although it may not have been modified here, I'm also confused by DefaultLogicalNodeManager's onNodeConfigurationChanged method. I see it calling the LifecycleSupervisor for each of the Sinks and Sources in the new configuration. What does it do with the Sinks and Sources from the previous configuration?
        Hide
        Arvind Prabhakar added a comment -

        Thanks for the feedback Ralph. As I pointed out earlier, this change only addresses the need to allow named instances. It does not address any other aspect of the system. The implementation of various components at this stage lacks a clear thread-safety strategy. We have touched upon this issue in various discussions before and I agree that we need to nail this down before the next release. The only reason I modified some methods to add a synchronized modifier in front was to make it consistent with the other methods in the class where I felt it was not. For what its worth, some subsystems such as JDBC channel provider are fully thread safe and have concurrent tests that verify the correctness in multi-threaded scenario.

        In order to address the concurrency requirements, we need to first cut over the Guava service implementation, and then model the components with it's state cycle to allow for concurrent access. Adopting this service framework will also iron out issues such as #4 that you have pointed out (which I think should be a clean shutdown of previous components before creating new ones).

        Show
        Arvind Prabhakar added a comment - Thanks for the feedback Ralph. As I pointed out earlier, this change only addresses the need to allow named instances. It does not address any other aspect of the system. The implementation of various components at this stage lacks a clear thread-safety strategy. We have touched upon this issue in various discussions before and I agree that we need to nail this down before the next release. The only reason I modified some methods to add a synchronized modifier in front was to make it consistent with the other methods in the class where I felt it was not. For what its worth, some subsystems such as JDBC channel provider are fully thread safe and have concurrent tests that verify the correctness in multi-threaded scenario. In order to address the concurrency requirements, we need to first cut over the Guava service implementation, and then model the components with it's state cycle to allow for concurrent access. Adopting this service framework will also iron out issues such as #4 that you have pointed out (which I think should be a clean shutdown of previous components before creating new ones).
        Arvind Prabhakar made changes -
        Affects Version/s v1.0.0 [ 12318896 ]
        Hide
        Arvind Prabhakar added a comment -

        As the discussion has tapered off I am going to mark this issue resolved. If any further issue needs to be addressed as a result of this, please open new Jiras.

        Show
        Arvind Prabhakar added a comment - As the discussion has tapered off I am going to mark this issue resolved. If any further issue needs to be addressed as a result of this, please open new Jiras.
        Hide
        Arvind Prabhakar added a comment -

        Patch committed to flume-728 branch.

        Show
        Arvind Prabhakar added a comment - Patch committed to flume-728 branch.
        Arvind Prabhakar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s v1.1.0 [ 12319284 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development