Flume
  1. Flume
  2. FLUME-1074

AvroSink if any non-caught exception is thrown, an exception is thrown in finally clause

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.1.0
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      If AvroSink and perhaps others, if any non-caught exception is thrown before transaction.commit, the finally clause throws:

      2012-03-28 17:31:24,560 ERROR org.apache.flume.SinkRunner: Unhandled exception, logging and sleeping for 5000ms
      java.lang.IllegalStateException: close() called when transaction is OPEN - you must either commit or rollback first
      at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
      at org.apache.flume.channel.BasicTransactionSemantics.close(BasicTransactionSemantics.java:179)
      at org.apache.flume.sink.AvroSink.process(AvroSink.java:269)
      at org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:118)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:148)
      at java.lang.Thread.run(Thread.java:662)

      1. FLUME-1074.patch
        1 kB
        Prasad Mujumdar

        Activity

        Hide
        Hudson added a comment -

        Integrated in flume-trunk #154 (See https://builds.apache.org/job/flume-trunk/154/)
        FLUME-1074. Fix exception handling in AvroSink.

        (Prasad Mujumdar via Arvind Prabhakar) (Revision 1308197)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1308197
        Files :

        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java
        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
        Show
        Hudson added a comment - Integrated in flume-trunk #154 (See https://builds.apache.org/job/flume-trunk/154/ ) FLUME-1074 . Fix exception handling in AvroSink. (Prasad Mujumdar via Arvind Prabhakar) (Revision 1308197) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1308197 Files : /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Prasad!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Prasad!
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4568/#review6578
        -----------------------------------------------------------

        Ship it!

        +1

        Thanks for the patch Prasad, please attach it to the Jira.

        • Arvind

        On 2012-03-30 05:33:54, Prasad Mujumdar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4568/

        -----------------------------------------------------------

        (Updated 2012-03-30 05:33:54)

        Review request for Flume, Arvind Prabhakar and Mike Percy.

        Summary

        -------

        Add exception handlers to rollback open transactions in error cases.

        This addresses bug FLUME-1074.

        https://issues.apache.org/jira/browse/FLUME-1074

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32

        flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8

        Diff: https://reviews.apache.org/r/4568/diff

        Testing

        -------

        regression test run

        Thanks,

        Prasad

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4568/#review6578 ----------------------------------------------------------- Ship it! +1 Thanks for the patch Prasad, please attach it to the Jira. Arvind On 2012-03-30 05:33:54, Prasad Mujumdar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4568/ ----------------------------------------------------------- (Updated 2012-03-30 05:33:54) Review request for Flume, Arvind Prabhakar and Mike Percy. Summary ------- Add exception handlers to rollback open transactions in error cases. This addresses bug FLUME-1074 . https://issues.apache.org/jira/browse/FLUME-1074 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8 Diff: https://reviews.apache.org/r/4568/diff Testing ------- regression test run Thanks, Prasad
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4568/#review6569
        -----------------------------------------------------------

        Prasad, good catch. These are the only two places with a Transaction.commit() and without a catch (Exception e) clause.

        Looks good to me.

        • Mike

        On 2012-03-30 05:33:54, Prasad Mujumdar wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4568/

        -----------------------------------------------------------

        (Updated 2012-03-30 05:33:54)

        Review request for Flume, Arvind Prabhakar and Mike Percy.

        Summary

        -------

        Add exception handlers to rollback open transactions in error cases.

        This addresses bug FLUME-1074.

        https://issues.apache.org/jira/browse/FLUME-1074

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32

        flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8

        Diff: https://reviews.apache.org/r/4568/diff

        Testing

        -------

        regression test run

        Thanks,

        Prasad

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4568/#review6569 ----------------------------------------------------------- Prasad, good catch. These are the only two places with a Transaction.commit() and without a catch (Exception e) clause. Looks good to me. Mike On 2012-03-30 05:33:54, Prasad Mujumdar wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4568/ ----------------------------------------------------------- (Updated 2012-03-30 05:33:54) Review request for Flume, Arvind Prabhakar and Mike Percy. Summary ------- Add exception handlers to rollback open transactions in error cases. This addresses bug FLUME-1074 . https://issues.apache.org/jira/browse/FLUME-1074 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8 Diff: https://reviews.apache.org/r/4568/diff Testing ------- regression test run Thanks, Prasad
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4568/
        -----------------------------------------------------------

        Review request for Flume, Arvind Prabhakar and Mike Percy.

        Summary
        -------

        Add exception handlers to rollback open transactions in error cases.

        This addresses bug FLUME-1074.
        https://issues.apache.org/jira/browse/FLUME-1074

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32
        flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8

        Diff: https://reviews.apache.org/r/4568/diff

        Testing
        -------

        regression test run

        Thanks,

        Prasad

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4568/ ----------------------------------------------------------- Review request for Flume, Arvind Prabhakar and Mike Percy. Summary ------- Add exception handlers to rollback open transactions in error cases. This addresses bug FLUME-1074 . https://issues.apache.org/jira/browse/FLUME-1074 Diffs flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java 7e24e32 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java e231cb8 Diff: https://reviews.apache.org/r/4568/diff Testing ------- regression test run Thanks, Prasad
        Hide
        Prasad Mujumdar added a comment -

        Also for sources via ChannelProcessor. I will include that in the patch.

        Show
        Prasad Mujumdar added a comment - Also for sources via ChannelProcessor. I will include that in the patch.
        Hide
        Brock Noland added a comment -

        This is also true for HDFSEvent sink if a non-Exception throwable is thrown.

        Show
        Brock Noland added a comment - This is also true for HDFSEvent sink if a non-Exception throwable is thrown.

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            Brock Noland
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development