Flume
  1. Flume
  2. FLUME-1606

Rollbacks of Put transactions does not clear the transaction from inflight puts.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.3.0
    • Component/s: None
    • Labels:
      None

      Description

      doRollback() calls queue.completeTransaction() only for take transactions. It should also be called for put transactions. A side effect of this is that the channel ends up thinking the data file which contains the rolled back transaction still has pending events, and will not delete any files which have ids after that one. Restarting the channel will cause files to be deleted if their ref count is zero, even if they are in inflight puts list. The reason the replay fixes this is that the inflight puts are not inserted into the queue because a commit for the puts is never seen. So this issue can be fixed by simply restarting the agent, at least until another put transaction is rolled back - which will result in the same issue.

      1. FLUME-1606.patch
        6 kB
        Hari Shreedharan

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          Patch that fixes this issue. Also has unit test which fails without this patch.

          Show
          Hari Shreedharan added a comment - Patch that fixes this issue. Also has unit test which fails without this patch.
          Hide
          Mike Percy added a comment -

          Patch committed. Thanks Hari!

          Rev: 79ff3adf0dfcd75b1a80261e6559a25eed1f3e08

          Show
          Mike Percy added a comment - Patch committed. Thanks Hari! Rev: 79ff3adf0dfcd75b1a80261e6559a25eed1f3e08
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #309 (See https://builds.apache.org/job/flume-trunk/309/)
          FLUME-1606. Rollbacks of Put transactions does not clear the transaction from inflight puts. (Revision 79ff3adf0dfcd75b1a80261e6559a25eed1f3e08)

          Result = SUCCESS
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=79ff3adf0dfcd75b1a80261e6559a25eed1f3e08
          Files :

          • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          Show
          Hudson added a comment - Integrated in flume-trunk #309 (See https://builds.apache.org/job/flume-trunk/309/ ) FLUME-1606 . Rollbacks of Put transactions does not clear the transaction from inflight puts. (Revision 79ff3adf0dfcd75b1a80261e6559a25eed1f3e08) Result = SUCCESS mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=79ff3adf0dfcd75b1a80261e6559a25eed1f3e08 Files : flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java

            People

            • Assignee:
              Hari Shreedharan
              Reporter:
              Hari Shreedharan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development