Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2401

Optionally Compress Backup Checkpoint

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0
    • Fix Version/s: 1.6.0
    • Component/s: File Channel
    • Labels:
      None

      Description

      File Channels with a large capacity create big checkpoint files. If the channel is mostly empty when the backup checkpoint is written the file will be large but mostly '0'. It would be nice to be able to optionally compress the checkpoint to reduce disk utilization for the backup checkpoint.

      See: http://mail-archives.apache.org/mod_mbox/flume-user/201406.mbox/%3C8B094F7E897248E89C8E55526178FD65%40cloudera.com%3E

      1. compress_backup_checkpoint_final.patch
        29 kB
        Abraham Fine
      2. compress_backup_checkpoint_new_tests.patch
        29 kB
        Abraham Fine
      3. compress_backup_checkpoint.patch
        26 kB
        Abraham Fine

        Activity

        Hide
        abrahamfine Abraham Fine added a comment -

        This patch includes all the functionality of the previous patch and additional tests recommended by Hari.

        • Original checkpoint is uncompressed, config changes to compress checkpoint - does the file channel restart from original checkpoint? are new checkpoints compressed?
        • Compressed checkpoint, config changes to not compress checkpoint - does channel start up? are new checkpoints uncompressed?
        Show
        abrahamfine Abraham Fine added a comment - This patch includes all the functionality of the previous patch and additional tests recommended by Hari. Original checkpoint is uncompressed, config changes to compress checkpoint - does the file channel restart from original checkpoint? are new checkpoints compressed? Compressed checkpoint, config changes to not compress checkpoint - does channel start up? are new checkpoints uncompressed?
        Hide
        hshreedharan Hari Shreedharan added a comment -

        HI Abe,

        Thanks for the patch! This looks good to go. I have only 2 minor comments:

        • the compressBackupCheckpoint field in Log class can be made final.
        • There are a lot of new/modified lines of code longer than 80 characters. Could you make sure that these lines are <= 80 chars long? Don't worry about old code, just the new lines you added (Eclipse/Intellij can do this for you)

        Once you have done this, please submit a new patch - I will commit it. Thanks!

        Show
        hshreedharan Hari Shreedharan added a comment - HI Abe, Thanks for the patch! This looks good to go. I have only 2 minor comments: the compressBackupCheckpoint field in Log class can be made final. There are a lot of new/modified lines of code longer than 80 characters. Could you make sure that these lines are <= 80 chars long? Don't worry about old code, just the new lines you added (Eclipse/Intellij can do this for you) Once you have done this, please submit a new patch - I will commit it. Thanks!
        Hide
        abrahamfine Abraham Fine added a comment -

        Hello Hari-

        Thank you for reviewing. Hopefully this patch addresses all of your concerns.

        Show
        abrahamfine Abraham Fine added a comment - Hello Hari- Thank you for reviewing. Hopefully this patch addresses all of your concerns.
        Hide
        hshreedharan Hari Shreedharan added a comment -

        +1. Committing this now.

        Show
        hshreedharan Hari Shreedharan added a comment - +1. Committing this now.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=69fd6b3 ]

        FLUME-2401. Optionally compress backup checkpoint.

        (Abraham Fine via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=69fd6b3 ] FLUME-2401 . Optionally compress backup checkpoint. (Abraham Fine via Hari Shreedharan)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 266f48b5fa77604ac36a0be4588863ba16b2d666 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=266f48b ]

        FLUME-2401. Optionally compress backup checkpoint.

        (Abraham Fine via Hari Shreedharan)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 266f48b5fa77604ac36a0be4588863ba16b2d666 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=266f48b ] FLUME-2401 . Optionally compress backup checkpoint. (Abraham Fine via Hari Shreedharan)
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Committed! Thanks Abraham!

        Show
        hshreedharan Hari Shreedharan added a comment - Committed! Thanks Abraham!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #645 (See https://builds.apache.org/job/flume-trunk/645/)
        FLUME-2401. Optionally compress backup checkpoint. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c)

        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
        • pom.xml
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
        • flume-ng-channels/flume-file-channel/pom.xml
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #645 (See https://builds.apache.org/job/flume-trunk/645/ ) FLUME-2401 . Optionally compress backup checkpoint. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c ) flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java pom.xml flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java flume-ng-channels/flume-file-channel/pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #5 (See https://builds.apache.org/job/Flume-trunk-hbase-98/5/)
        FLUME-2401. Optionally compress backup checkpoint. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c)

        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
        • pom.xml
        • flume-ng-channels/flume-file-channel/pom.xml
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java
        • flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java
        • flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #5 (See https://builds.apache.org/job/Flume-trunk-hbase-98/5/ ) FLUME-2401 . Optionally compress backup checkpoint. (harishreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=69fd6b3ad5e5b9ae6f1293b3d8e57ed57fd6701c ) flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelRestart.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java pom.xml flume-ng-channels/flume-file-channel/pom.xml flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFactory.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Serialization.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFileV3.java flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventQueueBackingStoreFile.java flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannelBase.java

          People

          • Assignee:
            abrahamfine Abraham Fine
            Reporter:
            abrahamfine Abraham Fine
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development