Flume
  1. Flume
  2. FLUME-1320

Add safeguard for checkpoint corruption detection

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      The checkpoint process can potentially be interrupted by a catastrophic failure such as complete power loss leaving the checkpoint corrupted. When the channel is restarted, the (corrupted) checkpoint will get loaded and there could be dataloss as a result of this.

      It is therefore necessary to have a safeguard to detect if a checkpoint succeeded or failed.

      1. FLUME-1320-4.patch
        28 kB
        Arvind Prabhakar

        Activity

        Hide
        Arvind Prabhakar added a comment -

        Attached a patch that addresses this issue.

        Show
        Arvind Prabhakar added a comment - Attached a patch that addresses this issue.
        Hide
        Mike Percy added a comment -

        Patch committed. Thanks Arvind!

        Show
        Mike Percy added a comment - Patch committed. Thanks Arvind!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #258 (See https://builds.apache.org/job/flume-trunk/258/)
        FLUME-1320. Add safeguard for checkpoint corruption detection.

        (Arvind Prabhakar via Mike Percy) (Revision 1353888)

        Result = UNSTABLE
        mpercy : http://svn.apache.org/viewvc/?view=rev&rev=1353888
        Files :

        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
        • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
        Show
        Hudson added a comment - Integrated in flume-trunk #258 (See https://builds.apache.org/job/flume-trunk/258/ ) FLUME-1320 . Add safeguard for checkpoint corruption detection. (Arvind Prabhakar via Mike Percy) (Revision 1353888) Result = UNSTABLE mpercy : http://svn.apache.org/viewvc/?view=rev&rev=1353888 Files : /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java

          People

          • Assignee:
            Arvind Prabhakar
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development