Flume
  1. Flume
  2. FLUME-820

JDBC channel should support capacity specification.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: NG alpha 2
    • Component/s: None
    • Labels:
      None

      Description

      users should be able to specify capacity for the JDBC channel in terms of total number of events stored. This will help ensure that the channel does not overwhelm the system by occupying the entire available disk space.

      1. FLUME-820-1.patch
        13 kB
        Arvind Prabhakar

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-728 #63 (See https://builds.apache.org/job/flume-728/63/)
        FLUME-820. Support for capacity specification for JDBC channel.

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1208691
        Files :

        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java
        • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
        Show
        Hudson added a comment - Integrated in flume-728 #63 (See https://builds.apache.org/job/flume-728/63/ ) FLUME-820 . Support for capacity specification for JDBC channel. arvind : http://svn.apache.org/viewvc/?view=rev&rev=1208691 Files : /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
        Hide
        Arvind Prabhakar added a comment -

        patch committed.

        Show
        Arvind Prabhakar added a comment - patch committed.
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-11-30 18:13:08, Eric Sammer wrote:

        > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java, line 1004

        > <https://reviews.apache.org/r/2937/diff/1/?file=60132#file60132line1004>

        >

        > Not sure if Statement implements Closable, but maybe use Guava Closeables.closeQuietly(o)?

        It does not, although it does have a close method. Such boilerplate code is an eyesore and can be moved/removed once Java starts supporting closures.

        • Arvind

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2937/#review3581
        -----------------------------------------------------------

        On 2011-11-26 21:14:07, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2937/

        -----------------------------------------------------------

        (Updated 2011-11-26 21:14:07)

        Review request for Flume and Prasad Mujumdar.

        Summary

        -------

        Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality.

        Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same.

        This addresses bug FLUME-820.

        https://issues.apache.org/jira/browse/FLUME-820

        Diffs

        -----

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d

        Diff: https://reviews.apache.org/r/2937/diff

        Testing

        -------

        Ran all JDBC channel tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-11-30 18:13:08, Eric Sammer wrote: > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java, line 1004 > < https://reviews.apache.org/r/2937/diff/1/?file=60132#file60132line1004 > > > Not sure if Statement implements Closable, but maybe use Guava Closeables.closeQuietly(o)? It does not, although it does have a close method. Such boilerplate code is an eyesore and can be moved/removed once Java starts supporting closures. Arvind ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2937/#review3581 ----------------------------------------------------------- On 2011-11-26 21:14:07, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2937/ ----------------------------------------------------------- (Updated 2011-11-26 21:14:07) Review request for Flume and Prasad Mujumdar. Summary ------- Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality. Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same. This addresses bug FLUME-820 . https://issues.apache.org/jira/browse/FLUME-820 Diffs ----- flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d Diff: https://reviews.apache.org/r/2937/diff Testing ------- Ran all JDBC channel tests. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2937/#review3581
        -----------------------------------------------------------

        Ship it!

        lgtm.

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java
        <https://reviews.apache.org/r/2937/#comment7994>

        Not sure if Statement implements Closable, but maybe use Guava Closeables.closeQuietly(o)?

        • Eric

        On 2011-11-26 21:14:07, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2937/

        -----------------------------------------------------------

        (Updated 2011-11-26 21:14:07)

        Review request for Flume and Prasad Mujumdar.

        Summary

        -------

        Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality.

        Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same.

        This addresses bug FLUME-820.

        https://issues.apache.org/jira/browse/FLUME-820

        Diffs

        -----

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9

        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16

        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d

        Diff: https://reviews.apache.org/r/2937/diff

        Testing

        -------

        Ran all JDBC channel tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2937/#review3581 ----------------------------------------------------------- Ship it! lgtm. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java < https://reviews.apache.org/r/2937/#comment7994 > Not sure if Statement implements Closable, but maybe use Guava Closeables.closeQuietly(o)? Eric On 2011-11-26 21:14:07, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2937/ ----------------------------------------------------------- (Updated 2011-11-26 21:14:07) Review request for Flume and Prasad Mujumdar. Summary ------- Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality. Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same. This addresses bug FLUME-820 . https://issues.apache.org/jira/browse/FLUME-820 Diffs ----- flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d Diff: https://reviews.apache.org/r/2937/diff Testing ------- Ran all JDBC channel tests. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2937/
        -----------------------------------------------------------

        Review request for Flume and Prasad Mujumdar.

        Summary
        -------

        Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality.

        Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same.

        This addresses bug FLUME-820.
        https://issues.apache.org/jira/browse/FLUME-820

        Diffs


        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9
        flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9
        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51
        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16
        flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d

        Diff: https://reviews.apache.org/r/2937/diff

        Testing
        -------

        Ran all JDBC channel tests.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2937/ ----------------------------------------------------------- Review request for Flume and Prasad Mujumdar. Summary ------- Added support for specifying the capacity of the JDBC channel. If a non-zero positive value is specified, the channel enforces it as the limit on total number of events persisted at any time. If a zero/negative/unspecified value is provided, the channel operates without any limit. Added a test case as well to exercise this functionality. Another change that I am sliding in is reducing the overall number of events that tests exercise - from about 2000 to close to 200. This will greatly reduce the amount of time the tests take when running on the JDBC channel module. The test functionality remains the same. This addresses bug FLUME-820 . https://issues.apache.org/jira/browse/FLUME-820 Diffs flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/ConfigurationConstants.java 7e0b3ac flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/DerbySchemaHandler.java 7b29c7d flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java c1bc400 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/MySQLSchemaHandler.java 1065ef9 flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/SchemaHandler.java 68946c9 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java 78eda51 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestDerbySchemaHandlerQueries.java 9ce6d16 flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 4c3012d Diff: https://reviews.apache.org/r/2937/diff Testing ------- Ran all JDBC channel tests. Thanks, Arvind

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development