[FLUME-617] ZK backed multimaster 2.0

• Type: Epic
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.9.4
• Fix Version/s: 0.9.5
• Component/s: Master
• Assignee: Unassigned
• Reporter: Jonathan Hsieh
• Created: 25/Apr/11 14:13
• Updated: 05/Nov/14 10:14
• Resolved: 05/Nov/14 10:14

Description

Multi master configurations for high availability and node scalability suffers from several deficiencies. This jira will serve as a coordination point for another implementation of the master's configuration storage mechanism and or a possible re-architecting of is code.

These are some of the core issues:

1) Translations for logical nodes (auto*Chain, logicalSource/Sink) does not work in mutli-master.
2) Flows abstraction is not scalable.
3) ZK-backend needs to be revisited to be scalable (single znodes vs several znodes).
4) OOME error occurs due to ack sharing mechanism.
5) Each master maintains its own command history.

Fundamentally, this is due to two core issues:

1) Not all state information is shared.
2) Current implementation does not have a leader

Issue Links

• blocks: FLUME-675 autoCollectorSource and auto*Chain/Sinks fail when master is restarted.
• blocks: FLUME-674 Flows 2.0
• blocks: FLUME-194 masters need to share additional state information for advanced translation features
• is blocked by: FLUME-114 Multi Masters not syncing updated configs properly
• is blocked by: FLUME-381 Configs from peer masters are not translated
• is related to: FLUME-52 Heartbeat code still uses timestamp rather than logical clock to decide if config is up-to-date