Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: Channel
    • Labels:
      None

      Description

      FLUME-896 turned into a durable Memory Channel, we need a durable File Channel

      1. FLUME-1085-7.patch
        177 kB
        Brock Noland

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          Patch is on RB

          Show
          Brock Noland added a comment - Patch is on RB
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          Review request for Flume.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs


          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/pom.xml 926e58a
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6723
          -----------------------------------------------------------

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment14577>

          break

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java
          <https://reviews.apache.org/r/4661/#comment14581>

          does not need to be serializable

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment14582>

          Preconditions check

          • Brock

          On 2012-04-05 18:59:03, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-05 18:59:03)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6723 ----------------------------------------------------------- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment14577 > break flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java < https://reviews.apache.org/r/4661/#comment14581 > does not need to be serializable flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment14582 > Preconditions check Brock On 2012-04-05 18:59:03, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-05 18:59:03) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6739
          -----------------------------------------------------------

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
          <https://reviews.apache.org/r/4661/#comment14725>

          What if a commit is in a previous file than the puts? We should round robin based on transaction id so all events for a transaction are in the same file.

          • Brock

          On 2012-04-05 18:59:03, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-05 18:59:03)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6739 ----------------------------------------------------------- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java < https://reviews.apache.org/r/4661/#comment14725 > What if a commit is in a previous file than the puts? We should round robin based on transaction id so all events for a transaction are in the same file. Brock On 2012-04-05 18:59:03, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-05 18:59:03) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6740
          -----------------------------------------------------------

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
          <https://reviews.apache.org/r/4661/#comment14726>

          It's possible a file could be created but we are unable to write the header. Currently this will throw an EOFException in the constructor and the user would have to catch this and ignore it. It's probably preferable that the user is allowed to call next which returns null since the file is empty.

          • Brock

          On 2012-04-05 18:59:03, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-05 18:59:03)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6740 ----------------------------------------------------------- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java < https://reviews.apache.org/r/4661/#comment14726 > It's possible a file could be created but we are unable to write the header. Currently this will throw an EOFException in the constructor and the user would have to catch this and ignore it. It's probably preferable that the user is allowed to call next which returns null since the file is empty. Brock On 2012-04-05 18:59:03, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-05 18:59:03) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6741
          -----------------------------------------------------------

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java
          <https://reviews.apache.org/r/4661/#comment14727>

          Remove trailing whitespace on non-final-channel files edited.

          • Brock

          On 2012-04-05 18:59:03, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-05 18:59:03)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6741 ----------------------------------------------------------- flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java < https://reviews.apache.org/r/4661/#comment14727 > Remove trailing whitespace on non-final-channel files edited. Brock On 2012-04-05 18:59:03, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-05 18:59:03) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-04-06 22:02:33.549650)

          Review request for Flume.

          Changes
          -------

          Latest patch addresses my comments and adds a few tests.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml 926e58a
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-06 22:02:33.549650) Review request for Flume. Changes ------- Latest patch addresses my comments and adds a few tests. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-06 13:53:29, Brock Noland wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java, line 260

          > <https://reviews.apache.org/r/4661/diff/2/?file=100325#file100325line260>

          >

          > It's possible a file could be created but we are unable to write the header. Currently this will throw an EOFException in the constructor and the user would have to catch this and ignore it. It's probably preferable that the user is allowed to call next which returns null since the file is empty.

          This EOFException is declared and then caught by the caller since the object has getters that are invalid if the header is not read.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6740
          -----------------------------------------------------------

          On 2012-04-06 22:02:33, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-06 22:02:33)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-06 13:53:29, Brock Noland wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java, line 260 > < https://reviews.apache.org/r/4661/diff/2/?file=100325#file100325line260 > > > It's possible a file could be created but we are unable to write the header. Currently this will throw an EOFException in the constructor and the user would have to catch this and ignore it. It's probably preferable that the user is allowed to call next which returns null since the file is empty. This EOFException is declared and then caught by the caller since the object has getters that are invalid if the header is not read. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6740 ----------------------------------------------------------- On 2012-04-06 22:02:33, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-06 22:02:33) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-04-08 22:34:48.594621)

          Review request for Flume.

          Changes
          -------

          There were some races in how Log cleaned up logs. We now find the smallest file id from the queue and open for write. We then ignore all files greater than this id. Additionally we insure we have at least two data files per data directory. Added integration test to TestFileChannel.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml 926e58a
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-08 22:34:48.594621) Review request for Flume. Changes ------- There were some races in how Log cleaned up logs. We now find the smallest file id from the queue and open for write. We then ignore all files greater than this id. Additionally we insure we have at least two data files per data directory. Added integration test to TestFileChannel. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6783
          -----------------------------------------------------------

          noting a few small items

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15003>

          Check closed flag

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15004>

          Check closed flag

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15005>

          Set<Integer>

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15006>

          Test with mockito

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java
          <https://reviews.apache.org/r/4661/#comment15007>

          UTF8

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java
          <https://reviews.apache.org/r/4661/#comment15008>

          UTF8

          • Brock

          On 2012-04-08 22:34:48, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-08 22:34:48)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6783 ----------------------------------------------------------- noting a few small items flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15003 > Check closed flag flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15004 > Check closed flag flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15005 > Set<Integer> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15006 > Test with mockito flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java < https://reviews.apache.org/r/4661/#comment15007 > UTF8 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java < https://reviews.apache.org/r/4661/#comment15008 > UTF8 Brock On 2012-04-08 22:34:48, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-08 22:34:48) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-04-09 00:37:34.744709)

          Review request for Flume.

          Changes
          -------

          Addressed latest patch. This is ready for external review so I will leave this along for now.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml 926e58a
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-09 00:37:34.744709) Review request for Flume. Changes ------- Addressed latest patch. This is ready for external review so I will leave this along for now. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6913
          -----------------------------------------------------------

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15362>

          Need to release permits during rollback of put

          • Brock

          On 2012-04-09 00:37:34, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-09 00:37:34)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6913 ----------------------------------------------------------- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15362 > Need to release permits during rollback of put Brock On 2012-04-09 00:37:34, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-09 00:37:34) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6921
          -----------------------------------------------------------

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15403>

          We should perhaps block for a very small period of time like MemoryChannel.

          • Brock

          On 2012-04-09 00:37:34, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-09 00:37:34)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml 926e58a

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6921 ----------------------------------------------------------- flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15403 > We should perhaps block for a very small period of time like MemoryChannel. Brock On 2012-04-09 00:37:34, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-09 00:37:34) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml 926e58a flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-04-17 06:36:33.519636)

          Review request for Flume.

          Changes
          -------

          Updated patch based on two comments below. Also added two tests, one each for restart and reconfiguration.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml e8155be
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-17 06:36:33.519636) Review request for Flume. Changes ------- Updated patch based on two comments below. Also added two tests, one each for restart and reconfiguration. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          Thanks for the patch Brock. The implementation looks great to me! Some feedback follows:

          Some convention nits:

          • Please make sure the line lengths are under 80 chars.
          • Please use all caps for variables that are static final.

          One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration.

          If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15053>

          Please use space instead of tab.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15054>

          The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15055>

          It would be better to add the shutdown hook in the start() method instead. Also, in the stop() the shutdown hook should be removed to avoid spurious exceptions.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15090>

          wish: it would be better to have this as a thread local to safeguard against transaction leaks.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15056>

          an IllegalStateException with a message indicating the scope of the damage/corruption would be appropriate.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15057>

          considering this class is called Log, it may be better to call this member LOGGER or DEBUG etc.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15058>

          Suggest using public accessor.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          <https://reviews.apache.org/r/4661/#comment15059>

          Please use enums instead.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          <https://reviews.apache.org/r/4661/#comment15060>

          Better to use IllegalStateException with a descriptive message.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          <https://reviews.apache.org/r/4661/#comment15061>

          Better to use IllegalStateException with descriptive message.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          <https://reviews.apache.org/r/4661/#comment15576>

          Please remove tab.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          <https://reviews.apache.org/r/4661/#comment15608>

          Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed.

          The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          <https://reviews.apache.org/r/4661/#comment15607>

          From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0.

          One way to address this would be modulo the converted value before return:

          return (next + index % elements.capacity()) % elements.capacity();

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15574>

          To work out of the box, this directory would need to be created if it does not exist. That would be similar to the behavior of JDBC channel as well.

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          <https://reviews.apache.org/r/4661/#comment15575>

          Same as above - will need to create the directory if it does not exist.

          • Arvind

          On 2012-04-17 06:36:33, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-17 06:36:33)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- Thanks for the patch Brock. The implementation looks great to me! Some feedback follows: Some convention nits: Please make sure the line lengths are under 80 chars. Please use all caps for variables that are static final. One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration. If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15053 > Please use space instead of tab. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15054 > The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15055 > It would be better to add the shutdown hook in the start() method instead. Also, in the stop() the shutdown hook should be removed to avoid spurious exceptions. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15090 > wish: it would be better to have this as a thread local to safeguard against transaction leaks. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15056 > an IllegalStateException with a message indicating the scope of the damage/corruption would be appropriate. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15057 > considering this class is called Log, it may be better to call this member LOGGER or DEBUG etc. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15058 > Suggest using public accessor. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java < https://reviews.apache.org/r/4661/#comment15059 > Please use enums instead. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java < https://reviews.apache.org/r/4661/#comment15060 > Better to use IllegalStateException with a descriptive message. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java < https://reviews.apache.org/r/4661/#comment15061 > Better to use IllegalStateException with descriptive message. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java < https://reviews.apache.org/r/4661/#comment15576 > Please remove tab. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java < https://reviews.apache.org/r/4661/#comment15608 > Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed. The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java < https://reviews.apache.org/r/4661/#comment15607 > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0. One way to address this would be modulo the converted value before return: return (next + index % elements.capacity()) % elements.capacity(); flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15574 > To work out of the box, this directory would need to be created if it does not exist. That would be similar to the behavior of JDBC channel as well. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java < https://reviews.apache.org/r/4661/#comment15575 > Same as above - will need to create the directory if it does not exist. Arvind On 2012-04-17 06:36:33, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-17 06:36:33) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review7022
          -----------------------------------------------------------

          One comment that did not get included in my previous feedback:

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
          <https://reviews.apache.org/r/4661/#comment15609>

          This test has a race condition with the background worker thread that cleans up the files. In order for this test to be deterministic, you may want to expose the shutdownWorker() method from log that can be used for this run to ensure that no files get deleted.

          • Arvind

          On 2012-04-17 06:36:33, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-17 06:36:33)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review7022 ----------------------------------------------------------- One comment that did not get included in my previous feedback: flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java < https://reviews.apache.org/r/4661/#comment15609 > This test has a race condition with the background worker thread that cleans up the files. In order for this test to be deterministic, you may want to expose the shutdownWorker() method from log that can be used for this run to ensure that no files get deleted. Arvind On 2012-04-17 06:36:33, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-17 06:36:33) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214

          > <https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214>

          >

          > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0.

          >

          > One way to address this would be modulo the converted value before return:

          >

          > return (next + index % elements.capacity()) % elements.capacity();

          >

          I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-17 06:36:33, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-17 06:36:33)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214 > < https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214 > > > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0. > > One way to address this would be modulo the converted value before return: > > return (next + index % elements.capacity()) % elements.capacity(); > I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-17 06:36:33, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-17 06:36:33) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          Brock Noland added a comment -

          FLUME-1121 has changes which call Channel.start()

          Show
          Brock Noland added a comment - FLUME-1121 has changes which call Channel.start()
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53.168303)

          Review request for Flume.

          Changes
          -------

          Latest patch addresses most of the concerns. Will comment on them individually as needed.

          Additionally, this patch is not meant for commit. It contains fixes for

          FLUME-1131
          FLUME-1132
          FLUME-1133

          and some debug things I am using to do manual integration testing.

          Also note, that this patch depends on FLUME-1121

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml e8155be
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e
          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851
          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53.168303) Review request for Flume. Changes ------- Latest patch addresses most of the concerns. Will comment on them individually as needed. Additionally, this patch is not meant for commit. It contains fixes for FLUME-1131 FLUME-1132 FLUME-1133 and some debug things I am using to do manual integration testing. Also note, that this patch depends on FLUME-1121 Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows:

          >

          > Some convention nits:

          > * Please make sure the line lengths are under 80 chars.

          > * Please use all caps for variables that are static final.

          >

          > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration.

          >

          > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent.

          >

          >

          I added locks to ensure only one channel has access to the directories at a given time.

          This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach.

          Thoughts?

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214

          > <https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214>

          >

          > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0.

          >

          > One way to address this would be modulo the converted value before return:

          >

          > return (next + index % elements.capacity()) % elements.capacity();

          >

          Brock Noland wrote:

          I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it.

          I have never written a data structure like this one, so I encourage a thorough review of it.

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 46

          > <https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line46>

          >

          > Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed.

          >

          > The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance.

          I added a note, if we find this is troublesome we do a later JIRA. Since we will be writing to disk every time we do modulus I think it's unlikely to be a performance problem.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows: > > Some convention nits: > * Please make sure the line lengths are under 80 chars. > * Please use all caps for variables that are static final. > > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration. > > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent. > > I added locks to ensure only one channel has access to the directories at a given time. This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach. Thoughts? On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214 > < https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214 > > > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0. > > One way to address this would be modulo the converted value before return: > > return (next + index % elements.capacity()) % elements.capacity(); > Brock Noland wrote: I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it. I have never written a data structure like this one, so I encourage a thorough review of it. On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 46 > < https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line46 > > > Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed. > > The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance. I added a note, if we find this is troublesome we do a later JIRA. Since we will be writing to disk every time we do modulus I think it's unlikely to be a performance problem. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 198

          > <https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line198>

          >

          > The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources.

          Agreed, note that the channel's start method is currently not called. FLUME-1121 addresses this.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 198 > < https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line198 > > > The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources. Agreed, note that the channel's start method is currently not called. FLUME-1121 addresses this. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, lines 289-290

          > <https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line289>

          >

          > wish: it would be better to have this as a thread local to safeguard against transaction leaks.

          This was addressed in the latest patch. However, I wonder if we should be doing this as BasicChannelSemantics.getTransaction() already does and users cannot call createTransaction() themselves.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, lines 289-290 > < https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line289 > > > wish: it would be better to have this as a thread local to safeguard against transaction leaks. This was addressed in the latest patch. However, I wonder if we should be doing this as BasicChannelSemantics.getTransaction() already does and users cannot call createTransaction() themselves. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows:

          >

          > Some convention nits:

          > * Please make sure the line lengths are under 80 chars.

          > * Please use all caps for variables that are static final.

          >

          > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration.

          >

          > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent.

          >

          >

          Brock Noland wrote:

          I added locks to ensure only one channel has access to the directories at a given time.

          This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach.

          Thoughts?

          For JDBC channel, this is a necessary evil. If the channel was not multi-tenant capable, it would mean a new connection pool, statement caches, cursors, etc for every channel instance. In which case the agent would become extremely resource heavy and would require extensive configuration to work out of the box. So instead we chose to use the channel name to disambiguate. I agree that this needs to be clearly documented and that users should know that the events will be lost if they change the channel name. The good news is that this is recoverable and even manually fixable.

          Since the file channel does not suffer from similare heavy resource requirements, it is good to have locks to ensure there is no corruption; even though that means that for an agent only one channel instance can exist with default configuration.

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 198

          > <https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line198>

          >

          > The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources.

          Brock Noland wrote:

          Agreed, note that the channel's start method is currently not called. FLUME-1121 addresses this.

          Thanks for pointing that out. I will prioritize the review of 1121 to get that in before this patch.

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, lines 289-290

          > <https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line289>

          >

          > wish: it would be better to have this as a thread local to safeguard against transaction leaks.

          Brock Noland wrote:

          This was addressed in the latest patch. However, I wonder if we should be doing this as BasicChannelSemantics.getTransaction() already does and users cannot call createTransaction() themselves.

          Agreed. Makes more sense to do it in the BasicChannelSemantics implementation. We should perhaps file another Jira just for that work, and during its implementation revert the logic you have added in your patch to avoid unnecessary duplication.

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 46

          > <https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line46>

          >

          > Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed.

          >

          > The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance.

          Brock Noland wrote:

          I added a note, if we find this is troublesome we do a later JIRA. Since we will be writing to disk every time we do modulus I think it's unlikely to be a performance problem.

          Makes sense.

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214

          > <https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214>

          >

          > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0.

          >

          > One way to address this would be modulo the converted value before return:

          >

          > return (next + index % elements.capacity()) % elements.capacity();

          >

          Brock Noland wrote:

          I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it.

          Brock Noland wrote:

          I have never written a data structure like this one, so I encourage a thorough review of it.

          Will do my best. Although, I have never reviewed a data structure like this before too Regardless, if there are issues we can always do follow-up jiras.

          • Arvind

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows: > > Some convention nits: > * Please make sure the line lengths are under 80 chars. > * Please use all caps for variables that are static final. > > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration. > > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent. > > Brock Noland wrote: I added locks to ensure only one channel has access to the directories at a given time. This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach. Thoughts? For JDBC channel, this is a necessary evil. If the channel was not multi-tenant capable, it would mean a new connection pool, statement caches, cursors, etc for every channel instance. In which case the agent would become extremely resource heavy and would require extensive configuration to work out of the box. So instead we chose to use the channel name to disambiguate. I agree that this needs to be clearly documented and that users should know that the events will be lost if they change the channel name. The good news is that this is recoverable and even manually fixable. Since the file channel does not suffer from similare heavy resource requirements, it is good to have locks to ensure there is no corruption; even though that means that for an agent only one channel instance can exist with default configuration. On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, line 198 > < https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line198 > > > The initialization of the Log instance along with replaying it should be done in the start() method. Correspondingly, the stop() method should shutdown the log and release all associated resources. Brock Noland wrote: Agreed, note that the channel's start method is currently not called. FLUME-1121 addresses this. Thanks for pointing that out. I will prioritize the review of 1121 to get that in before this patch. On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java, lines 289-290 > < https://reviews.apache.org/r/4661/diff/5/?file=101000#file101000line289 > > > wish: it would be better to have this as a thread local to safeguard against transaction leaks. Brock Noland wrote: This was addressed in the latest patch. However, I wonder if we should be doing this as BasicChannelSemantics.getTransaction() already does and users cannot call createTransaction() themselves. Agreed. Makes more sense to do it in the BasicChannelSemantics implementation. We should perhaps file another Jira just for that work, and during its implementation revert the logic you have added in your patch to avoid unnecessary duplication. On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 46 > < https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line46 > > > Not a request, but something to keep an eye on: this implementation uses the modulo operation heavily to calculate the queue positions. The alternative way of doing this would be to maintain head, size and optionally tail pointers. That will reduce the number of overall modulo operations needed. > > The only reason I point this out is that modulo is CPU intensive for large numbers and therefore may become a concern for performance. Brock Noland wrote: I added a note, if we find this is troublesome we do a later JIRA. Since we will be writing to disk every time we do modulus I think it's unlikely to be a performance problem. Makes sense. On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java, line 214 > < https://reviews.apache.org/r/4661/diff/6/?file=102330#file102330line214 > > > From cursory analysis it seems that this will likely not be able to handle the wrap-around logic correctly. For example, if the capacity is 10, next is 9 and size/index is 1: the calculated index will be 10, when it should be 0. > > One way to address this would be modulo the converted value before return: > > return (next + index % elements.capacity()) % elements.capacity(); > Brock Noland wrote: I am working on the rest of the feedback, but I just wanted to say "big thank you" for finding this bug. I hit it last night while stress testing and was planning on spending much of my day finding it. Brock Noland wrote: I have never written a data structure like this one, so I encourage a thorough review of it. Will do my best. Although, I have never reviewed a data structure like this before too Regardless, if there are issues we can always do follow-up jiras. Arvind ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-19 09:12:40, Arvind Prabhakar wrote:

          > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows:

          >

          > Some convention nits:

          > * Please make sure the line lengths are under 80 chars.

          > * Please use all caps for variables that are static final.

          >

          > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration.

          >

          > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent.

          >

          >

          Brock Noland wrote:

          I added locks to ensure only one channel has access to the directories at a given time.

          This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach.

          Thoughts?

          Arvind Prabhakar wrote:

          For JDBC channel, this is a necessary evil. If the channel was not multi-tenant capable, it would mean a new connection pool, statement caches, cursors, etc for every channel instance. In which case the agent would become extremely resource heavy and would require extensive configuration to work out of the box. So instead we chose to use the channel name to disambiguate. I agree that this needs to be clearly documented and that users should know that the events will be lost if they change the channel name. The good news is that this is recoverable and even manually fixable.

          Since the file channel does not suffer from similare heavy resource requirements, it is good to have locks to ensure there is no corruption; even though that means that for an agent only one channel instance can exist with default configuration.

          Makes sense to me.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review6786
          -----------------------------------------------------------

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-19 09:12:40, Arvind Prabhakar wrote: > Thanks for the patch Brock. The implementation looks great to me! Some feedback follows: > > Some convention nits: > * Please make sure the line lengths are under 80 chars. > * Please use all caps for variables that are static final. > > One high-level consideration: Since it is possible to have multiple file channels within the same agent, we should either make file channel multi-tenant capable like the jdbc channel, or instead have the ability to associate directory locks for the file channel instance for checkpoint and data directories. Doing this will ensure that there is no corruption and the system exits cleanly in case of misconfiguration. > > If you chose to implement the non-multitenant system, it will be preferable to use the channel name within the default paths for checkpoint and data directories. This will ensure minimum configuration necessary even when using multiple file channels within the same agent. > > Brock Noland wrote: I added locks to ensure only one channel has access to the directories at a given time. This is a concern I have a about JDBC Channel. I am not sold on using the channel name from the configuration as that is a symbolic name, users will likely think they can change it. However, if there are events stored in that channel and they change the name, all their events are gone. If we used the name this same thing would occur. Today with JDBC Channel and this implementation of FileChannel, it can be multi-tenant if configured to use different directories for each Channel. I feel like this is a better approach. Thoughts? Arvind Prabhakar wrote: For JDBC channel, this is a necessary evil. If the channel was not multi-tenant capable, it would mean a new connection pool, statement caches, cursors, etc for every channel instance. In which case the agent would become extremely resource heavy and would require extensive configuration to work out of the box. So instead we chose to use the channel name to disambiguate. I agree that this needs to be clearly documented and that users should know that the events will be lost if they change the channel name. The good news is that this is recoverable and even manually fixable. Since the file channel does not suffer from similare heavy resource requirements, it is good to have locks to ensure there is no corruption; even though that means that for an agent only one channel instance can exist with default configuration. Makes sense to me. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review6786 ----------------------------------------------------------- On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review7574
          -----------------------------------------------------------

          Hi Brock, any chance you could update this patch for commit?

          • Arvind

          On 2012-04-19 20:10:53, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-04-19 20:10:53)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e

          flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8

          flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review7574 ----------------------------------------------------------- Hi Brock, any chance you could update this patch for commit? Arvind On 2012-04-19 20:10:53, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-04-19 20:10:53) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/resources/log4j.properties 739ecc8 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-core/src/main/java/org/apache/flume/channel/ChannelProcessor.java eb6460e flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8 flume-ng-core/src/main/java/org/apache/flume/sink/NullSink.java c812851 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          Brock Noland added a comment -

          Patch from RB

          Show
          Brock Noland added a comment - Patch from RB
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/
          -----------------------------------------------------------

          (Updated 2012-05-05 16:34:23.320371)

          Review request for Flume.

          Changes
          -------

          I addressed the review items and removed items from other JIRAs.

          Summary
          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.
          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs (updated)


          flume-ng-channels/flume-file-channel/pom.xml e8155be
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION
          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java f42cee8
          flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java d3aa316
          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION
          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4661/diff

          Testing
          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-05-05 16:34:23.320371) Review request for Flume. Changes ------- I addressed the review items and removed items from other JIRAs. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs (updated) flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java f42cee8 flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java d3aa316 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4661/#review7619
          -----------------------------------------------------------

          Ship it!

          +1

          • Arvind

          On 2012-05-05 16:34:23, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4661/

          -----------------------------------------------------------

          (Updated 2012-05-05 16:34:23)

          Review request for Flume.

          Summary

          -------

          This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs.

          This addresses bug FLUME-1085.

          https://issues.apache.org/jira/browse/FLUME-1085

          Diffs

          -----

          flume-ng-channels/flume-file-channel/pom.xml e8155be

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION

          flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java f42cee8

          flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java d3aa316

          flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION

          flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION

          Diff: https://reviews.apache.org/r/4661/diff

          Testing

          -------

          Unit tests and integration tests added to cover obvious cases.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/#review7619 ----------------------------------------------------------- Ship it! +1 Arvind On 2012-05-05 16:34:23, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4661/ ----------------------------------------------------------- (Updated 2012-05-05 16:34:23) Review request for Flume. Summary ------- This patch implements a durable file channel. It does by writing all transaction events to disk and syncing to disk when a commit occurs. It does have a memory component in that pointers to the event on disk are kept in memory. This will consume 8 bytes of direct memory (non-heap) per event. Some basic calculations are in the FileChannel java docs. This addresses bug FLUME-1085 . https://issues.apache.org/jira/browse/FLUME-1085 Diffs ----- flume-ng-channels/flume-file-channel/pom.xml e8155be flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453 flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java PRE-CREATION flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java ab66998 flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java PRE-CREATION flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java f42cee8 flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java d3aa316 flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java PRE-CREATION flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java PRE-CREATION Diff: https://reviews.apache.org/r/4661/diff Testing ------- Unit tests and integration tests added to cover obvious cases. Thanks, Brock
          Hide
          Arvind Prabhakar added a comment -

          Patch committed. Thanks Brock!

          Show
          Arvind Prabhakar added a comment - Patch committed. Thanks Brock!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #196 (See https://builds.apache.org/job/flume-trunk/196/)
          FLUME-1085. Implement a durable File Channel.

          (Brock Noland via Arvind Prabhakar) (Revision 1334488)

          Result = SUCCESS
          arvind : http://svn.apache.org/viewvc/?view=rev&rev=1334488
          Files :

          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/pom.xml
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
          • /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java
          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java
          • /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java
          Show
          Hudson added a comment - Integrated in flume-trunk #196 (See https://builds.apache.org/job/flume-trunk/196/ ) FLUME-1085 . Implement a durable File Channel. (Brock Noland via Arvind Prabhakar) (Revision 1334488) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1334488 Files : /incubator/flume/trunk/flume-ng-channels/flume-file-channel/pom.xml /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEvent.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventPointer.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogUtils.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Pair.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSinkRunner.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/CountingSourceRunner.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEvent.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventPointer.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java /incubator/flume/trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/StressSource.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/tools/DirectMemoryUtils.java
          Hide
          Mubarak Seyed added a comment -

          @Brock
          I have specified -XX:MaxDirectMemorySize=16M but i am getting

          INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory(null)
          

          Am i missing any libs/parameters? Thanks.

          Show
          Mubarak Seyed added a comment - @Brock I have specified -XX:MaxDirectMemorySize=16M but i am getting INFO tools.DirectMemoryUtils: Unable to get maxDirectMemory from VM: NoSuchMethodException: sun.misc.VM.maxDirectMemory( null ) Am i missing any libs/parameters? Thanks.
          Hide
          Brock Noland added a comment -

          @Mubarak,

          This is probably best for the flume-user list since this JIRA's are for development not troubleshooting. However, that is a normal, see it's INFO level, your DirectMemorySize parameter is still being respected.

          Brock

          Show
          Brock Noland added a comment - @Mubarak, This is probably best for the flume-user list since this JIRA's are for development not troubleshooting. However, that is a normal, see it's INFO level, your DirectMemorySize parameter is still being respected. Brock

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development