Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2259

transaction closure not happening for all the scenario in hbasesink

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.5.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Hbase sink will not close the transaction if any error occurs like below

      06 Dec 2013 12:04:50,466 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver event. Exception follows.
      org.apache.flume.ChannelException: Failed to obtain lock for writing to the log. Try increasing the log write timeout value. [channel=c1]
      	at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:512)
      	at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
      	at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
      	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:239)
      	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      	at java.lang.Thread.run(Thread.java:662)
      

      This will cause hbase sink to become non operative.

      java.lang.IllegalStateException: begin() called when transaction is OPEN!
      	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
      	at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
      	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:236)
      	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      	at java.lang.Thread.run(Thread.java:662)
      04 Dec 2013 16:55:38,553 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver event. Exception follows.
      java.lang.IllegalStateException: begin() called when transaction is OPEN!
      	at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
      	at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
      	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:236)
      	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      	at java.lang.Thread.run(Thread.java:662)
      
      1. FLUME-2259_1.patch
        4 kB
        Gopinathan A
      2. FLUME-2259.patch
        2 kB
        Gopinathan A
      3. FLUME-2259-2.patch
        10 kB
        Gopinathan A

        Issue Links

          Activity

          Hide
          gopinathan.av Gopinathan A added a comment -

          I will upload patch..

          Show
          gopinathan.av Gopinathan A added a comment - I will upload patch..
          Hide
          hshreedharan Hari Shreedharan added a comment -

          I have noticed this too. This is because the transaction handling is in a different method and if the serializer or channel throws, then the transaction is not rolled back.

          Show
          hshreedharan Hari Shreedharan added a comment - I have noticed this too. This is because the transaction handling is in a different method and if the serializer or channel throws, then the transaction is not rolled back.
          Hide
          gopinathan.av Gopinathan A added a comment -

          Yes. I will move that logic to outer method.

          Show
          gopinathan.av Gopinathan A added a comment - Yes. I will move that logic to outer method.
          Hide
          gopinathan.av Gopinathan A added a comment -

          I have not handled serializer error scenario in this patch. I will update the patch once its done.

          Show
          gopinathan.av Gopinathan A added a comment - I have not handled serializer error scenario in this patch. I will update the patch once its done.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Gopinathan A - This looks good in general. There is a small problem with the patch though. It is possible that the tx.close gets called without a tx.rollback if a failure happens in the try block and not in putEventsAndCommit method. For example, if channel.take() or serializer.getActions() throws an exception. We must handle this case. One way of doing this is by not doing the rollback in the putEventsAndCommit method, instead allowing the exception thrown in that method to get thrown into the process method where we catch it and rollback and then close.

          Show
          hshreedharan Hari Shreedharan added a comment - Gopinathan A - This looks good in general. There is a small problem with the patch though. It is possible that the tx.close gets called without a tx.rollback if a failure happens in the try block and not in putEventsAndCommit method. For example, if channel.take() or serializer.getActions() throws an exception. We must handle this case. One way of doing this is by not doing the rollback in the putEventsAndCommit method, instead allowing the exception thrown in that method to get thrown into the process method where we catch it and rollback and then close.
          Hide
          gopinathan.av Gopinathan A added a comment -

          Thanks Hari Shreedharan

          Yes. I will update the patch

          Show
          gopinathan.av Gopinathan A added a comment - Thanks Hari Shreedharan Yes. I will update the patch
          Hide
          gopinathan.av Gopinathan A added a comment -

          Please review the updated patch.

          Show
          gopinathan.av Gopinathan A added a comment - Please review the updated patch.
          Hide
          paliwalashish Ashish Paliwal added a comment -

          Please open a review request on https://reviews.apache.org

          More info at https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute
          It has patch naming convention as well

          Show
          paliwalashish Ashish Paliwal added a comment - Please open a review request on https://reviews.apache.org More info at https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute It has patch naming convention as well
          Hide
          gopinathan.av Gopinathan A added a comment -

          Added testcases.

          Please review this patch.

          Show
          gopinathan.av Gopinathan A added a comment - Added testcases. Please review this patch.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          +1. This looks good. Committing.

          Show
          hshreedharan Hari Shreedharan added a comment - +1. This looks good. Committing.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          I am making a couple minor changes before committing:

          • Backoff is returned only if batch is empty.
          • moved txn.begin() to try block.
          • Fixed a bunch of typos.
          Show
          hshreedharan Hari Shreedharan added a comment - I am making a couple minor changes before committing: Backoff is returned only if batch is empty. moved txn.begin() to try block. Fixed a bunch of typos.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit cf2ac371351812f333960604a3a8ad4a510307ae in branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=cf2ac37 ]

          FLUME-2259. Transaction closure not happening for all the scenario in HBaseSink.

          (Gopinathan A via Hari Shreedharan)

          Show
          jira-bot ASF subversion and git services added a comment - Commit cf2ac371351812f333960604a3a8ad4a510307ae in branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=cf2ac37 ] FLUME-2259 . Transaction closure not happening for all the scenario in HBaseSink. (Gopinathan A via Hari Shreedharan)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ee75bc57cdacf98620035ff0acb132821c01c9bb in branch refs/heads/flume-1.5 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ee75bc5 ]

          FLUME-2259. Transaction closure not happening for all the scenario in HBaseSink.

          (Gopinathan A via Hari Shreedharan)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ee75bc57cdacf98620035ff0acb132821c01c9bb in branch refs/heads/flume-1.5 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ee75bc5 ] FLUME-2259 . Transaction closure not happening for all the scenario in HBaseSink. (Gopinathan A via Hari Shreedharan)
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Committed. Thanks Gopinathan!

          Show
          hshreedharan Hari Shreedharan added a comment - Committed. Thanks Gopinathan!
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in flume-trunk #538 (See https://builds.apache.org/job/flume-trunk/538/)
          FLUME-2259. Transaction closure not happening for all the scenario in HBaseSink. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cf2ac371351812f333960604a3a8ad4a510307ae)

          • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestHBaseSink.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/MockSimpleHbaseEventSerializer.java
          • flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #538 (See https://builds.apache.org/job/flume-trunk/538/ ) FLUME-2259 . Transaction closure not happening for all the scenario in HBaseSink. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=cf2ac371351812f333960604a3a8ad4a510307ae ) flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestHBaseSink.java flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/MockSimpleHbaseEventSerializer.java flume-ng-sinks/flume-ng-hbase-sink/src/main/java/org/apache/flume/sink/hbase/HBaseSink.java
          Hide
          gopinathan.av Gopinathan A added a comment -

          Thanks Hari

          Show
          gopinathan.av Gopinathan A added a comment - Thanks Hari

            People

            • Assignee:
              gopinathan.av Gopinathan A
              Reporter:
              gopinathan.av Gopinathan A
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development