Flume
  1. Flume
  2. FLUME-1586

File Channel should support verifying integrity of individual events.

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.4.0
    • Component/s: Channel
    • Labels:
      None
    1. FLUME-1586-2.patch
      202 kB
      Hari Shreedharan
    2. FLUME-1586-2-rebased.patch
      201 kB
      Hari Shreedharan

      Issue Links

        Activity

        Hide
        Hari Shreedharan added a comment -

        I uploaded a skeleton implementation for this jira on reviewboard. There is some code which needs to be filled in, which should answer the question:

        • What to do when the checksums do not match?

        To make sure we don't hit performance too much, comparison of checksums is done only during replay. As of now, this is what is done:

        • The Put TransactionEventRecord header now has an optional checksum field. (As of now, the code to enable checksumming has not been added).
        • During replay, the checksum of the event body is compared to the checksum in the header. What to do if this does not match - TBD.
        Show
        Hari Shreedharan added a comment - I uploaded a skeleton implementation for this jira on reviewboard. There is some code which needs to be filled in, which should answer the question: What to do when the checksums do not match? To make sure we don't hit performance too much, comparison of checksums is done only during replay. As of now, this is what is done: The Put TransactionEventRecord header now has an optional checksum field. (As of now, the code to enable checksumming has not been added). During replay, the checksum of the event body is compared to the checksum in the header. What to do if this does not match - TBD.
        Hide
        Hari Shreedharan added a comment -

        This patch cannot address corruption which extends across event boundaries. This can handle only event body being corrupted. This can be extended to checksum the event headers too, but that I am leaving it to a future jira to address that.

        Show
        Hari Shreedharan added a comment - This patch cannot address corruption which extends across event boundaries. This can handle only event body being corrupted. This can be extended to checksum the event headers too, but that I am leaving it to a future jira to address that.
        Hide
        Hari Shreedharan added a comment -

        Added code to enable checksumming through configuration, and uploaded new patch to rb.

        Show
        Hari Shreedharan added a comment - Added code to enable checksumming through configuration, and uploaded new patch to rb.
        Hide
        Brock Noland added a comment -

        Marking as a duplicate of FLUME-1485.

        Show
        Brock Noland added a comment - Marking as a duplicate of FLUME-1485 .
        Hide
        Hari Shreedharan added a comment -

        I am working on a slightly different approach to this one, and adding a tool to clear our corrupt events after backing up the file.

        Show
        Hari Shreedharan added a comment - I am working on a slightly different approach to this one, and adding a tool to clear our corrupt events after backing up the file.
        Hide
        Hari Shreedharan added a comment -

        One of the things I am not doing for now is support for encrypted channels. I believe that can be added later. Does that sound fair?

        Show
        Hari Shreedharan added a comment - One of the things I am not doing for now is support for encrypted channels. I believe that can be added later. Does that sound fair?
        Hide
        Hari Shreedharan added a comment -

        Please see review board for initial patch.

        Show
        Hari Shreedharan added a comment - Please see review board for initial patch.
        Hide
        Brock Noland added a comment -

        Hari Shreedharan can you post the patch on this JIRA?

        Show
        Brock Noland added a comment - Hari Shreedharan can you post the patch on this JIRA?
        Hide
        Hari Shreedharan added a comment -

        Patch from rb.

        Show
        Hari Shreedharan added a comment - Patch from rb.
        Hide
        Hari Shreedharan added a comment -

        Rebased patch

        Show
        Hari Shreedharan added a comment - Rebased patch
        Hide
        Brock Noland added a comment -

        Committed! Thanks Hari!

        Show
        Brock Noland added a comment - Committed! Thanks Hari!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #423 (See https://builds.apache.org/job/flume-trunk/423/)
        FLUME-1586: File Channel should support verifying integrity of individual events. (Revision c57ebd1d2296fd1c0c8f84eecb16799de2a73e35)

        Result = SUCCESS
        brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c57ebd1d2296fd1c0c8f84eecb16799de2a73e35
        Files :

        • flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java
        • flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java
        • pom.xml
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
        • flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java
        • flume-tools/pom.xml
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
        • flume-ng-dist/src/main/assembly/src.xml
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java
        • flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java
        • bin/flume-ng
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
        • flume-ng-dist/pom.xml
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
        • flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java
        • flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java
        • flume-ng-dist/src/main/assembly/bin.xml
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
        Show
        Hudson added a comment - Integrated in flume-trunk #423 (See https://builds.apache.org/job/flume-trunk/423/ ) FLUME-1586 : File Channel should support verifying integrity of individual events. (Revision c57ebd1d2296fd1c0c8f84eecb16799de2a73e35) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c57ebd1d2296fd1c0c8f84eecb16799de2a73e35 Files : flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java flume-tools/pom.xml flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java flume-ng-dist/src/main/assembly/src.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java bin/flume-ng flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java flume-ng-dist/pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java flume-ng-dist/src/main/assembly/bin.xml flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Hari Shreedharan
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development