Flume
  1. Flume
  2. FLUME-822

JDBC channel lock acquisition failure during take()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: NG alpha 1
    • Fix Version/s: NG alpha 2
    • Component/s: Channel
    • Labels:
      None

      Description

      The jdbc channel implementation (using Derby) results in a lock acquisition error upon take().

      It appears as if the event is successfully inserted by the source, but the sink fails to receive the event in the allotted time. The configuration used was as follows.

      ==
      localhost.sources = avro nc
      localhost.sinks = logger log2
      localhost.channels = ch1 ch2

      localhost.channels.ch1.type = memory
      localhost.channels.ch1.capacity = 10000

      localhost.sources.avro.type = avro
      localhost.sources.avro.bind = 0.0.0.0
      localhost.sources.avro.port = 41414
      localhost.sources.avro.channels = ch1

      localhost.sinks.logger.type = logger
      localhost.sinks.logger.channel = ch1

      #

      localhost.channels.ch2.type = jdbc
      localhost.sources.nc.type = netcat
      localhost.sources.nc.bind = 0.0.0.0
      localhost.sources.nc.port = 41415
      localhost.sources.nc.name = nc
      localhost.sources.nc.channels = ch2
      localhost.sinks.log2.type = logger
      localhost.sinks.log2.channel = ch2
      ==

      The path in use is nc(netcat) -> ch2(jdbc) -> log2(logger) with minimal configuration. I'll attach log output shortly.

      1. flume-jdbc-lock-failure.log
        47 kB
        E. Sammer
      2. FLUME-822-1.patch
        3 kB
        Arvind Prabhakar

        Activity

        Hide
        E. Sammer added a comment -

        Attaching log with lock errors.

        Show
        E. Sammer added a comment - Attaching log with lock errors.
        Hide
        Arvind Prabhakar added a comment -

        So far I am unable to reproduce this issue. However, I will continue to test different scenarios to see if this happens in other cases.

        Show
        Arvind Prabhakar added a comment - So far I am unable to reproduce this issue. However, I will continue to test different scenarios to see if this happens in other cases.
        Hide
        Arvind Prabhakar added a comment -

        I have been looking through the code and found one place in the NetcatSource where there is an apparent transaction leak. This could be something that may have caused this problem. I will work on a general clean up patch for transaction handling and put it up for review soon.

        Show
        Arvind Prabhakar added a comment - I have been looking through the code and found one place in the NetcatSource where there is an apparent transaction leak. This could be something that may have caused this problem. I will work on a general clean up patch for transaction handling and put it up for review soon.
        Hide
        Arvind Prabhakar added a comment -

        This patch addresses some transaction related code that could potentially lead to lock problems and transaction leaks.

        Show
        Arvind Prabhakar added a comment - This patch addresses some transaction related code that could potentially lead to lock problems and transaction leaks.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2943/
        -----------------------------------------------------------

        Review request for Flume and Eric Sammer.

        Summary
        -------

        This change cleans up some code in LoggerSink, RollingFileSink and NetcatSource dealing with transaction handling.

        The problem highlighted in FLUME-822 is not reproducible so far and my hope is that this clean up code will help address any isolated cases that could cause it.

        This addresses bug FLUME-822.
        https://issues.apache.org/jira/browse/FLUME-822

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java f098604
        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 5632b75
        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java f9251a6

        Diff: https://reviews.apache.org/r/2943/diff

        Testing
        -------

        Ran all the tests successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2943/ ----------------------------------------------------------- Review request for Flume and Eric Sammer. Summary ------- This change cleans up some code in LoggerSink, RollingFileSink and NetcatSource dealing with transaction handling. The problem highlighted in FLUME-822 is not reproducible so far and my hope is that this clean up code will help address any isolated cases that could cause it. This addresses bug FLUME-822 . https://issues.apache.org/jira/browse/FLUME-822 Diffs flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java f098604 flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 5632b75 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java f9251a6 Diff: https://reviews.apache.org/r/2943/diff Testing ------- Ran all the tests successfully. Thanks, Arvind
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/2943/#review3578
        -----------------------------------------------------------

        Ship it!

        I'm embarrassed I never got back around to fixing these things. lgtm.

        • Eric

        On 2011-11-27 06:52:44, Arvind Prabhakar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/2943/

        -----------------------------------------------------------

        (Updated 2011-11-27 06:52:44)

        Review request for Flume and Eric Sammer.

        Summary

        -------

        This change cleans up some code in LoggerSink, RollingFileSink and NetcatSource dealing with transaction handling.

        The problem highlighted in FLUME-822 is not reproducible so far and my hope is that this clean up code will help address any isolated cases that could cause it.

        This addresses bug FLUME-822.

        https://issues.apache.org/jira/browse/FLUME-822

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java f098604

        flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 5632b75

        flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java f9251a6

        Diff: https://reviews.apache.org/r/2943/diff

        Testing

        -------

        Ran all the tests successfully.

        Thanks,

        Arvind

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2943/#review3578 ----------------------------------------------------------- Ship it! I'm embarrassed I never got back around to fixing these things. lgtm. Eric On 2011-11-27 06:52:44, Arvind Prabhakar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2943/ ----------------------------------------------------------- (Updated 2011-11-27 06:52:44) Review request for Flume and Eric Sammer. Summary ------- This change cleans up some code in LoggerSink, RollingFileSink and NetcatSource dealing with transaction handling. The problem highlighted in FLUME-822 is not reproducible so far and my hope is that this clean up code will help address any isolated cases that could cause it. This addresses bug FLUME-822 . https://issues.apache.org/jira/browse/FLUME-822 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java f098604 flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 5632b75 flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java f9251a6 Diff: https://reviews.apache.org/r/2943/diff Testing ------- Ran all the tests successfully. Thanks, Arvind
        Hide
        Arvind Prabhakar added a comment -

        Patch committed.

        Show
        Arvind Prabhakar added a comment - Patch committed.
        Hide
        Hudson added a comment -

        Integrated in flume-728 #63 (See https://builds.apache.org/job/flume-728/63/)
        FLUME-822. Transaction code cleanup to fix potential lock timeout issues.

        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1208645
        Files :

        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
        • /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
        Show
        Hudson added a comment - Integrated in flume-728 #63 (See https://builds.apache.org/job/flume-728/63/ ) FLUME-822 . Transaction code cleanup to fix potential lock timeout issues. arvind : http://svn.apache.org/viewvc/?view=rev&rev=1208645 Files : /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java /incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development