Details
-
Brainstorming
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
None
-
None
-
None
Description
Implementing the failover sink runner the following was suggested:
1. This needs to be implemented on top of FLUME-949 which deals with removing the notion of a PollableSink altogether. As a result, the SinkRunner will become a concrete implementation that can then allow different sink handling policies - such as either a failover policy (needed for this issue), or load balancing policy (not needed for this issue). Hence the policy part needs to be pluggable rather than the sink runner itself. An example of such a construct is the ChannelSelector and ChannelProcessor implementations.
In Flume-865 I have implemented FailoverSinkRunner as a separate runner, but I am open to the idea of making it pluggable if it makes the code more maintainable.
As is, there are many differences between the requirements for Failover and a normal Sink runner, including configuration, initialisation, shutdown, error handling and event processing. If we were to make this pluggable, many hooks would be needed and I don't think there is that much common behavior that warrants using a pluggable system rather than just a solid base class.
- Adding a new sink to a runner, with configuration variables(such as priority or weight)
- Policy for handling process: should this just return a list of sinks to process like ChannelSelector and hand off the processing to Process? I think that the specific failover policy for each type of runner will be different so this feels awkward. I would personally prefer to just pass the process call to the pluggable component and let it be responsible for calling process on the correct sinks, as well as handling errors.
Right now I am not convinced for the need to make SinkRunner pluggable, but I would be interested to hear other peoples opinions
Attachments
Issue Links
- is required by
-
FLUME-865 Implement failover sink
-
- Resolved
-