[FLUME-728] Flume NG refactoring

• Type: Epic
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: NG alpha 1
• Fix Version/s: NG alpha 1
• Component/s: Build, Docs, Master, Node, Shell, Sinks+Sources, Technical Debt, Test, Web
• Labels: NG
• Assignee: Eric Sammer
• Reporter: Eric Sammer
• Created: 09/Aug/11 06:24
• Updated: 22/Oct/11 20:58
• Resolved: 22/Oct/11 20:58

Description

To solve certain known issues and limitations, Flume requires a refactoring of some core classes and systems. This bug is a parent issue to track the development of a "Flume NG" - a poorly named, but necessary refactoring. Subtasks should be added to track individual systems and components.

The following known issues are specifically to be addressed:

• Code complexity; Flume has evolved over the last few years and has a fair amount of extraneous code.
• Core component lifecycle standardization and control code (e.g. anything that can be start()ed or stop()ed, sources, sinks).
• (Static) Configuration access throughout the code base.
• Drastic simplification of common data paths (e.g. durability as an element of the source rather than a disconnected sink).
• Heartbeat and master rearchitecture.
• Renaming packages to org.apache.flume.

This is a large and far reaching set of tasks. The intent is to perform this work in a branch as to not disrupt immediate releases or short term forthcoming releases while still allowing open development in the community.

Sub-Tasks

• Create NG internal diagrams: Resolved
• Create NG developer code tour docs: Resolved
• Define NG core interfaces and classes: Resolved
• Define NG node interfaces and classes: Resolved
• Define NG master interfaces and classes: Resolved
• Create NG WAL component: Resolved
• Port Flume OG sources and sinks to NG interfaces: Closed
• Create NG system configuration component: Resolved
• Create NG node configuration components: Closed
• Create NG command line launchers and daemon infrastructure: Closed
• Create metric collection infrastructure: Closed
• Add support for monitoring APIs: Resolved
• Implement JDBC based channel implementation: Closed
• Implement HDFS Flume NG sink: Closed