Flume
  1. Flume
  2. FLUME-1121

Recoverable Memory Channel cannot recover data

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.1.0, v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Run an agent with a source, like seq with a recoverable memory channel and no sink. kill the process with ctrl-C. Run an agent now with no source, but logger sink and the same recoverable memory channel. I expected the events generated by the source earlier should be written to the sink, but no events with picked up.

        Issue Links

          Activity

          Hide
          Hari Shreedharan added a comment -

          Seems like this is because start() never gets called for a channel from the configuration provider implementations. They start only sources and sinks. Even if I somehow managed to start it, by adding start calls in the properties file config provider, there are other issues. The replay function tries to load all events into the internal memory channel, but if there are more than n events(n = size of memory channel), due to multiple previous failures or as a result of reconfiguration, the start() throws an exception and the remaining events in the backing file are not picked up. Also I used a seq source to generate some 107 events, with a channel of size 100, and no sinks. I expected that only 100 would go to the file, but the next time it starts up, it tries to load all 107 and fails and throws exceptions.

          Show
          Hari Shreedharan added a comment - Seems like this is because start() never gets called for a channel from the configuration provider implementations. They start only sources and sinks. Even if I somehow managed to start it, by adding start calls in the properties file config provider, there are other issues. The replay function tries to load all events into the internal memory channel, but if there are more than n events(n = size of memory channel), due to multiple previous failures or as a result of reconfiguration, the start() throws an exception and the remaining events in the backing file are not picked up. Also I used a seq source to generate some 107 events, with a channel of size 100, and no sinks. I expected that only 100 would go to the file, but the next time it starts up, it tries to load all 107 and fails and throws exceptions.
          Hide
          Brock Noland added a comment -

          Can you post your configuration? I am unable to more than 107 events on the channel.

          Show
          Brock Noland added a comment - Can you post your configuration? I am unable to more than 107 events on the channel.
          Hide
          Brock Noland added a comment -

          I was able to reproduce.

          Show
          Brock Noland added a comment - I was able to reproduce.
          Hide
          Hari Shreedharan added a comment -

          Brock:

          I noticed more than one problem. One of the issues was that even though the source got exceptions when trying to write more events than the capacity of the channel(N), the source got exceptions(there was no sink attached), for the extra events(K). Later when I removed the source, and simply attached a sink, then the replay function would create a transaction for the internal memory channel to load all events from the file. Though this should only have been N, it tries to load N+K events, and fails right away. In the above example, N was 100 and K was 7. K can be anything, the only thing that determines K is the time you stop sending events.

          I believe the replay function, and the way we load events from the file to memory channel need to be refactored. I think we should not accept new events from the source, until and unless we have replayed every event from the file and got commits from the sink for all those transactions. This way we can guarantee that all events will eventually be replayed and no events are lost(since the source would send failure back to the previous hop).

          That said, things seem to look OK if number of events put into the channel < total channel capacity.

          Show
          Hari Shreedharan added a comment - Brock: I noticed more than one problem. One of the issues was that even though the source got exceptions when trying to write more events than the capacity of the channel(N), the source got exceptions(there was no sink attached), for the extra events(K). Later when I removed the source, and simply attached a sink, then the replay function would create a transaction for the internal memory channel to load all events from the file. Though this should only have been N, it tries to load N+K events, and fails right away. In the above example, N was 100 and K was 7. K can be anything, the only thing that determines K is the time you stop sending events. I believe the replay function, and the way we load events from the file to memory channel need to be refactored. I think we should not accept new events from the source, until and unless we have replayed every event from the file and got commits from the sink for all those transactions. This way we can guarantee that all events will eventually be replayed and no events are lost(since the source would send failure back to the previous hop). That said, things seem to look OK if number of events put into the channel < total channel capacity.
          Hide
          Brock Noland added a comment -

          The patch I will be posting keeps track of capacity itself as we cannot rely on MemoryChannel's capacity checking because the semantics of RecoverableMemoryChannel are incompatibly different. Additionally I add a flag called open which is checked before put/take/commit operations.

          Show
          Brock Noland added a comment - The patch I will be posting keeps track of capacity itself as we cannot rely on MemoryChannel's capacity checking because the semantics of RecoverableMemoryChannel are incompatibly different. Additionally I add a flag called open which is checked before put/take/commit operations.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/
          -----------------------------------------------------------

          Review request for Flume.

          Summary
          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.
          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs


          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0
          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6
          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd
          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819
          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0
          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059
          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e
          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1
          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5
          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          Diff: https://reviews.apache.org/r/4713/diff

          Testing
          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          Hari Shreedharan added a comment - - edited

          I agree with the general changes, but I have a concern here. If the channel capacity is N, then only a total of N events should ever be there in the channel, regardless of failure. What I am saying is this:

          A) K events are committed to disk, and the agent fails. (K < N)
          B) J events are attempted to be written to the channel(assuming no sink or slow sink, which has not yet cleared any of the K events). If K+J > N, then only a total of N events should be in the channel(including the original K), and the rest(K+J-N) should return failure. Eg: K = 99, J = 5, then after the first of the 2nd lot of 5 events are written, the channel * should * fail.

          Else, the capacity of the channel is simply an implementation detail, and does not hold any real value. It is really the capacity of the memory channel held internally.

          Ideally, the moment there are more than N events in the channel (fail or not), the channel returns failure to append more events. As far as a client calling this is concerned, it does not matter how it is implemented internally, the channel should not have more than N events.

          Show
          Hari Shreedharan added a comment - - edited I agree with the general changes, but I have a concern here. If the channel capacity is N, then only a total of N events should ever be there in the channel, regardless of failure. What I am saying is this: A) K events are committed to disk, and the agent fails. (K < N) B) J events are attempted to be written to the channel(assuming no sink or slow sink, which has not yet cleared any of the K events). If K+J > N, then only a total of N events should be in the channel(including the original K), and the rest(K+J-N) should return failure. Eg: K = 99, J = 5, then after the first of the 2nd lot of 5 events are written, the channel * should * fail. Else, the capacity of the channel is simply an implementation detail, and does not hold any real value. It is really the capacity of the memory channel held internally. Ideally, the moment there are more than N events in the channel (fail or not), the channel returns failure to append more events. As far as a client calling this is concerned, it does not matter how it is implemented internally, the channel should not have more than N events.
          Hide
          Brock Noland added a comment -

          Are you saying with the changes you feel the capacity can be exceeded? Please describe how this can occur. From what I wrote, capacity is reserved everytime we do a put. Additionally the memory channel is repopulated before taking puts on restart.

          Show
          Brock Noland added a comment - Are you saying with the changes you feel the capacity can be exceeded? Please describe how this can occur. From what I wrote, capacity is reserved everytime we do a put. Additionally the memory channel is repopulated before taking puts on restart.
          Hide
          Hari Shreedharan added a comment -

          Apologies. I didn't look at your patch closely enough. Seems like you handle the above case, since you repopulate the memory channel before accepting more puts.

          But I am not sure if the reconfiguration case is handled correctly.

          I can have this case right:
          Capacity = 100.No sink, only one source.

          Write 11 events to channel, commit and then fail. - Capacity not exceeded.

          Reconfigure channel to 10 events. Now, the current code only logs error, but you'd still try to load all 11 to the memory channel.

          Otherwise, I think these changes look good.

          Show
          Hari Shreedharan added a comment - Apologies. I didn't look at your patch closely enough. Seems like you handle the above case, since you repopulate the memory channel before accepting more puts. But I am not sure if the reconfiguration case is handled correctly. I can have this case right: Capacity = 100.No sink, only one source. Write 11 events to channel, commit and then fail. - Capacity not exceeded. Reconfigure channel to 10 events. Now, the current code only logs error, but you'd still try to load all 11 to the memory channel. Otherwise, I think these changes look good.
          Hide
          Brock Noland added a comment -

          Yeah this is a tough thing to handle. I have changed the patch to throw an error when this occurs, the open flag will not be set and no activity will occur. I think this is valid since it's akin to lowering the heap on the NameNode to lower than what is required to run.

          Show
          Brock Noland added a comment - Yeah this is a tough thing to handle. I have changed the patch to throw an error when this occurs, the open flag will not be set and no activity will occur. I think this is valid since it's akin to lowering the heap on the NameNode to lower than what is required to run.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/
          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25.161684)

          Review request for Flume.

          Changes
          -------

          Added tests for restart and reconfiguration. Thrown an exception if the capacity is not sufficient on restart.

          Summary
          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.
          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs (updated)


          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449
          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6
          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc
          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5
          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e
          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd
          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b
          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819
          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0
          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73
          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27
          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34
          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing
          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25.161684) Review request for Flume. Changes ------- Added tests for restart and reconfiguration. Thrown an exception if the capacity is not sufficient on restart. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs (updated) flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          Hari Shreedharan added a comment -

          The workaround sounds good, at least for now. If the need be, we probably can revisit it later. Thanks Brock!

          Show
          Hari Shreedharan added a comment - The workaround sounds good, at least for now. If the need be, we probably can revisit it later. Thanks Brock!
          Hide
          Brock Noland added a comment -

          Can you do a review in RB?

          Show
          Brock Noland added a comment - Can you do a review in RB?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7380
          -----------------------------------------------------------

          Ship it!

          Thanks for the patch Brock, changes look good. Some minor feedback follows.

          Please rebase the patch when convenient and attach to the Jira.

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java
          <https://reviews.apache.org/r/4713/#comment16288>

          nit: this seems to be duplicating part of the functionality offered by the LifecycleState flag maintained by the abstract base class. It will be good to see if that can be reused here.

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java
          <https://reviews.apache.org/r/4713/#comment16284>

          nit:tab

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java
          <https://reviews.apache.org/r/4713/#comment16285>

          This call needs to follow the transaction idiom with the appropriate try/catch. Also, since this is initialization time code, it may be easier to do a bulk put in a single transaction.

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java
          <https://reviews.apache.org/r/4713/#comment16291>

          Since these are getting accessed by worker thread and no longer propagated, they should be marked volatile? (or propagated).

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java
          <https://reviews.apache.org/r/4713/#comment16283>

          The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well.

          • Arvind

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7380 ----------------------------------------------------------- Ship it! Thanks for the patch Brock, changes look good. Some minor feedback follows. Please rebase the patch when convenient and attach to the Jira. flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java < https://reviews.apache.org/r/4713/#comment16288 > nit: this seems to be duplicating part of the functionality offered by the LifecycleState flag maintained by the abstract base class. It will be good to see if that can be reused here. flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java < https://reviews.apache.org/r/4713/#comment16284 > nit:tab flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java < https://reviews.apache.org/r/4713/#comment16285 > This call needs to follow the transaction idiom with the appropriate try/catch. Also, since this is initialization time code, it may be easier to do a bulk put in a single transaction. flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java < https://reviews.apache.org/r/4713/#comment16291 > Since these are getting accessed by worker thread and no longer propagated, they should be marked volatile? (or propagated). flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java < https://reviews.apache.org/r/4713/#comment16283 > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well. Arvind On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7406
          -----------------------------------------------------------

          Hi Brock,

          Thanks for the patch! This is not really a review: I just have one feedback. Could you please not include the cleanup in this patch and submit a separate patch for that. The revision history will show this patch affecting every single file even though this is supposed to only fix the recoverable memory channel, and we will need to look at detailed revision history to look at what changes were caused by this patch. I'd rather see only the changes related to this bug.

          • Hari

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7406 ----------------------------------------------------------- Hi Brock, Thanks for the patch! This is not really a review: I just have one feedback. Could you please not include the cleanup in this patch and submit a separate patch for that. The revision history will show this patch affecting every single file even though this is supposed to only fix the recoverable memory channel, and we will need to look at detailed revision history to look at what changes were caused by this patch. I'd rather see only the changes related to this bug. Hari On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69

          > <https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60>

          >

          > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well.

          Who is executing sources now if PollableSourceRunning is not being used?

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7380
          -----------------------------------------------------------

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69 > < https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60 > > > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well. Who is executing sources now if PollableSourceRunning is not being used? Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7380 ----------------------------------------------------------- On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69

          > <https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60>

          >

          > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well.

          Brock Noland wrote:

          Who is executing sources now if PollableSourceRunning is not being used?

          Looks to me like it's still being used in PropertiesFileConfigurationProvider.loadSources?

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7380
          -----------------------------------------------------------

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69 > < https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60 > > > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well. Brock Noland wrote: Who is executing sources now if PollableSourceRunning is not being used? Looks to me like it's still being used in PropertiesFileConfigurationProvider.loadSources? Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7380 ----------------------------------------------------------- On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69

          > <https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60>

          >

          > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well.

          Brock Noland wrote:

          Who is executing sources now if PollableSourceRunning is not being used?

          Brock Noland wrote:

          Looks to me like it's still being used in PropertiesFileConfigurationProvider.loadSources?

          Brock, I stand corrected. It is still being used and not removed yet. Please leave that in as is. I will file a separate Jira to address that if it becomes critical.

          • Arvind

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7380
          -----------------------------------------------------------

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java, lines 60-69 > < https://reviews.apache.org/r/4713/diff/3/?file=101695#file101695line60 > > > The source runner interface is no longer used anywhere and should have been removed altogether (along with its subtypes). Since there is other cleanup in your patch, it would be great if you could remove that as well. Brock Noland wrote: Who is executing sources now if PollableSourceRunning is not being used? Brock Noland wrote: Looks to me like it's still being used in PropertiesFileConfigurationProvider.loadSources? Brock, I stand corrected. It is still being used and not removed yet. Please leave that in as is. I will file a separate Jira to address that if it becomes critical. Arvind ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7380 ----------------------------------------------------------- On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          Brock Noland added a comment -

          Patch from RB.

          Show
          Brock Noland added a comment - Patch from RB.
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > Thanks for the patch Brock, changes look good. Some minor feedback follows.

          >

          > Please rebase the patch when convenient and attach to the Jira.

          Thank you for your review! Attached to the JIRA!

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, line 123

          > <https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line123>

          >

          > nit:tab

          done

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, line 82

          > <https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line82>

          >

          > nit: this seems to be duplicating part of the functionality offered by the LifecycleState flag maintained by the abstract base class. It will be good to see if that can be reused here.

          I agree, I have created FLUME-1169 to implement this checking in the superclass.

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, lines 144-148

          > <https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line144>

          >

          > This call needs to follow the transaction idiom with the appropriate try/catch. Also, since this is initialization time code, it may be easier to do a bulk put in a single transaction.

          Done

          On 2012-04-30 19:22:02, Arvind Prabhakar wrote:

          > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java, lines 92-93

          > <https://reviews.apache.org/r/4713/diff/3/?file=101676#file101676line92>

          >

          > Since these are getting accessed by worker thread and no longer propagated, they should be marked volatile? (or propagated).

          I figured eventual consistency for this parameters was fine, however I have marked them volatile.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7380
          -----------------------------------------------------------

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > Thanks for the patch Brock, changes look good. Some minor feedback follows. > > Please rebase the patch when convenient and attach to the Jira. Thank you for your review! Attached to the JIRA! On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, line 123 > < https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line123 > > > nit:tab done On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, line 82 > < https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line82 > > > nit: this seems to be duplicating part of the functionality offered by the LifecycleState flag maintained by the abstract base class. It will be good to see if that can be reused here. I agree, I have created FLUME-1169 to implement this checking in the superclass. On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java, lines 144-148 > < https://reviews.apache.org/r/4713/diff/3/?file=101674#file101674line144 > > > This call needs to follow the transaction idiom with the appropriate try/catch. Also, since this is initialization time code, it may be easier to do a bulk put in a single transaction. Done On 2012-04-30 19:22:02, Arvind Prabhakar wrote: > flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java, lines 92-93 > < https://reviews.apache.org/r/4713/diff/3/?file=101676#file101676line92 > > > Since these are getting accessed by worker thread and no longer propagated, they should be marked volatile? (or propagated). I figured eventual consistency for this parameters was fine, however I have marked them volatile. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7380 ----------------------------------------------------------- On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2012-05-01 01:02:14, Hari Shreedharan wrote:

          > Hi Brock,

          >

          > Thanks for the patch! This is not really a review: I just have one feedback. Could you please not include the cleanup in this patch and submit a separate patch for that. The revision history will show this patch affecting every single file even though this is supposed to only fix the recoverable memory channel, and we will need to look at detailed revision history to look at what changes were caused by this patch. I'd rather see only the changes related to this bug.

          I was including these changes under the `Please fix any "broken windows" in your neighborhood' mantra from the `How to Contribute' section on the wiki. If we feel that changes like this should be done in separate JIRA's, lets update that item.

          • Brock

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/4713/#review7406
          -----------------------------------------------------------

          On 2012-04-13 20:12:25, Brock Noland wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/4713/

          -----------------------------------------------------------

          (Updated 2012-04-13 20:12:25)

          Review request for Flume.

          Summary

          -------

          Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager.

          Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory.

          Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future.

          This addresses bug FLUME-1121.

          https://issues.apache.org/jira/browse/FLUME-1121

          Diffs

          -----

          flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2

          flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1

          flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6

          flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc

          flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5

          flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e

          flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09

          flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd

          flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b

          flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b

          flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5

          flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819

          flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0

          flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73

          flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27

          flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34

          flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0

          Diff: https://reviews.apache.org/r/4713/diff

          Testing

          -------

          All unit tests pass and manual testing passes as well.

          Thanks,

          Brock

          Show
          jiraposter@reviews.apache.org added a comment - On 2012-05-01 01:02:14, Hari Shreedharan wrote: > Hi Brock, > > Thanks for the patch! This is not really a review: I just have one feedback. Could you please not include the cleanup in this patch and submit a separate patch for that. The revision history will show this patch affecting every single file even though this is supposed to only fix the recoverable memory channel, and we will need to look at detailed revision history to look at what changes were caused by this patch. I'd rather see only the changes related to this bug. I was including these changes under the `Please fix any "broken windows" in your neighborhood' mantra from the `How to Contribute' section on the wiki. If we feel that changes like this should be done in separate JIRA's, lets update that item. Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/#review7406 ----------------------------------------------------------- On 2012-04-13 20:12:25, Brock Noland wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4713/ ----------------------------------------------------------- (Updated 2012-04-13 20:12:25) Review request for Flume. Summary ------- Channel.start() is not being called. This is fixed in DefaultLogicalNodeManager. Additionally RecoverableMemoryChannel now tracks it's own capacity due to the MemoryChannel semantics being completely different. Basically, if we rely on MemoryChannel capacity, then an error will be thrown when we commit the MemoryChannelTransaction. However, we will have already committed this data to disk. If we commit to MemoryChannel first (there by checking capacity) we could fail to write to disk resulting in data which is only in memory. Also, I ran cleanup on modules touched. This removes whitespace, unused imports, and adds @Override tags where needed. This is one time cleanup which allows automated cleanup in the future. This addresses bug FLUME-1121 . https://issues.apache.org/jira/browse/FLUME-1121 Diffs ----- flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java 1421449 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2 flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26 flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1 flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 1f0e8c6 flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java 80020fc flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46 flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 732cce5 flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java ca5212e flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09 flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java b0485b1 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java 9d4a1fd flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java edd8a8b flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5 flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java 4722819 flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java f179de0 flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java 2334059 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java 97ef796 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java fa63b73 flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java 0622f27 flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java 07c3d0b flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java 225cd34 flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java 0b8a2c0 Diff: https://reviews.apache.org/r/4713/diff Testing ------- All unit tests pass and manual testing passes as well. Thanks, Brock
          Hide
          Arvind Prabhakar added a comment -

          Patch committed. Thanks Brock!

          Show
          Arvind Prabhakar added a comment - Patch committed. Thanks Brock!
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #190 (See https://builds.apache.org/job/flume-trunk/190/)
          FLUME-1121. Recoverable Memory Channel problem during recovery.

          (Brock Noland via Arvind Prabhakar) (Revision 1332885)

          Result = SUCCESS
          arvind : http://svn.apache.org/viewvc/?view=rev&rev=1332885
          Files :

          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java
          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java
          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java
          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java
          • /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java
          • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java
          • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java
          • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java
          • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java
          • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java
          • /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java
          • /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java
          • /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java
          • /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java
          Show
          Hudson added a comment - Integrated in flume-trunk #190 (See https://builds.apache.org/job/flume-trunk/190/ ) FLUME-1121 . Recoverable Memory Channel problem during recovery. (Brock Noland via Arvind Prabhakar) (Revision 1332885) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1332885 Files : /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/RecoverableMemoryChannel.java /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/SequenceIDBuffer.java /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WAL.java /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/main/java/org/apache/flume/channel/recoverable/memory/wal/WALDataFile.java /incubator/flume/trunk/flume-ng-channels/flume-recoverable-memory-channel/src/test/java/org/apache/flume/channel/recoverable/memory/TestRecoverableMemoryChannel.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/lifecycle/LifecycleAware.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AbstractSink.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestBasicChannelSemantics.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestChannelUtils.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java /incubator/flume/trunk/flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java /incubator/flume/trunk/flume-ng-node/src/main/java/org/apache/flume/node/nodemanager/DefaultLogicalNodeManager.java /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestFailoverRpcClient.java /incubator/flume/trunk/flume-ng-sdk/src/test/java/org/apache/flume/api/TestNettyAvroRpcClient.java

            People

            • Assignee:
              Brock Noland
              Reporter:
              Hari Shreedharan
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development