To solve certain known issues and limitations, Flume requires a refactoring of some core classes and systems. This bug is a parent issue to track the development of a "Flume NG" - a poorly named, but necessary refactoring. Subtasks should be added to track individual systems and components.
The following known issues are specifically to be addressed:
- Code complexity; Flume has evolved over the last few years and has a fair amount of extraneous code.
- Core component lifecycle standardization and control code (e.g. anything that can be start()ed or stop()ed, sources, sinks).
- (Static) Configuration access throughout the code base.
- Drastic simplification of common data paths (e.g. durability as an element of the source rather than a disconnected sink).
- Heartbeat and master rearchitecture.
- Renaming packages to org.apache.flume.
This is a large and far reaching set of tasks. The intent is to perform this work in a branch as to not disrupt immediate releases or short term forthcoming releases while still allowing open development in the community.
|1.||Create NG internal diagrams||Open|
|2.||Create NG developer code tour docs||Open|
|Define NG core interfaces and classes||Resolved|
|Define NG node interfaces and classes||Resolved|
|5.||Define NG master interfaces and classes||Open|
|Create NG WAL component||Resolved|
|Port Flume OG sources and sinks to NG interfaces||Closed|
|8.||Create NG system configuration component||Open||Unassigned|
|Create NG node configuration components||Closed|
|Create NG command line launchers and daemon infrastructure||Closed|
|Create metric collection infrastructure||Closed|
|12.||Add support for monitoring APIs||Open||Unassigned|
|Implement JDBC based channel implementation||Closed|
|Implement HDFS Flume NG sink||Closed|
Issues in Epic
|Field||Original Value||New Value|
|Affects Version/s||NG [ 12318440 ]|
|Fix Version/s||NG [ 12318440 ]|
|Status||Open [ 1 ]||Resolved [ 5 ]|
|Resolution||Fixed [ 1 ]|
|Status||Resolved [ 5 ]||Closed [ 6 ]|