I uploaded a skeleton implementation for this jira on reviewboard. There is some code which needs to be filled in, which should answer the question:

• What to do when the checksums do not match?

To make sure we don't hit performance too much, comparison of checksums is done only during replay. As of now, this is what is done:

• The Put TransactionEventRecord header now has an optional checksum field. (As of now, the code to enable checksumming has not been added).
• During replay, the checksum of the event body is compared to the checksum in the header. What to do if this does not match - TBD.

This patch cannot address corruption which extends across event boundaries. This can handle only event body being corrupted. This can be extended to checksum the event headers too, but that I am leaving it to a future jira to address that.

Added code to enable checksumming through configuration, and uploaded new patch to rb.

Marking as a duplicate of FLUME-1485.

I am working on a slightly different approach to this one, and adding a tool to clear our corrupt events after backing up the file.

One of the things I am not doing for now is support for encrypted channels. I believe that can be added later. Does that sound fair?

Please see review board for initial patch.

Hari Shreedharan can you post the patch on this JIRA?

Patch from rb.

Rebased patch

Committed! Thanks Hari!

Integrated in flume-trunk #423 (See https://builds.apache.org/job/flume-trunk/423/)
FLUME-1586: File Channel should support verifying integrity of individual events. (Revision c57ebd1d2296fd1c0c8f84eecb16799de2a73e35)

Result = SUCCESS
brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c57ebd1d2296fd1c0c8f84eecb16799de2a73e35
Files :

• flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java
• flume-tools/src/main/java/org/apache/flume/tools/FlumeToolsMain.java
• pom.xml
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
• flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java
• flume-tools/pom.xml
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/proto/ProtosFactory.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
• flume-ng-dist/src/main/assembly/src.xml
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CorruptEventException.java
• flume-tools/src/main/java/org/apache/flume/tools/FlumeTool.java
• bin/flume-ng
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
• flume-ng-dist/pom.xml
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/NoopRecordException.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
• flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java
• flume-tools/src/main/java/org/apache/flume/tools/FlumeToolType.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java
• flume-ng-dist/src/main/assembly/bin.xml
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
• flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
• flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java