Details

      Description

      To solve certain known issues and limitations, Flume requires a refactoring of some core classes and systems. This bug is a parent issue to track the development of a "Flume NG" - a poorly named, but necessary refactoring. Subtasks should be added to track individual systems and components.

      The following known issues are specifically to be addressed:

      • Code complexity; Flume has evolved over the last few years and has a fair amount of extraneous code.
      • Core component lifecycle standardization and control code (e.g. anything that can be start()ed or stop()ed, sources, sinks).
      • (Static) Configuration access throughout the code base.
      • Drastic simplification of common data paths (e.g. durability as an element of the source rather than a disconnected sink).
      • Heartbeat and master rearchitecture.
      • Renaming packages to org.apache.flume.

      This is a large and far reaching set of tasks. The intent is to perform this work in a branch as to not disrupt immediate releases or short term forthcoming releases while still allowing open development in the community.

        Issues in Epic

        There are no issues in this epic.

          Activity

          E. Sammer created issue -
          E. Sammer made changes -
          Field Original Value New Value
          Labels NG
          Hide
          Jonathan Hsieh added a comment -

          +1. Let's create a new branch for this development. Maybe call it 0.10-masterng?

          Show
          Jonathan Hsieh added a comment - +1. Let's create a new branch for this development. Maybe call it 0.10-masterng?
          Hide
          Jonathan Hsieh added a comment -

          Actually from reading up on svn repos, maybe we should create a 0.9.x branch for bug fixes, and do this on trunk/mainline instead.

          Show
          Jonathan Hsieh added a comment - Actually from reading up on svn repos, maybe we should create a 0.9.x branch for bug fixes, and do this on trunk/mainline instead.
          Hide
          Arvind Prabhakar added a comment -

          @Jon: I agree - once the 728 branch is stable and has sufficient functionality, it should be put on the trunk.

          Show
          Arvind Prabhakar added a comment - @Jon: I agree - once the 728 branch is stable and has sufficient functionality, it should be put on the trunk.
          E. Sammer made changes -
          Affects Version/s NG [ 12318440 ]
          E. Sammer made changes -
          Fix Version/s NG [ 12318440 ]
          Hide
          Hudson added a comment -

          Integrated in flume-728 #50 (See https://builds.apache.org/job/flume-728/50/)
          FLUME-728. Added a multi-threaded test to exercise JDBC channel.

          arvind : http://svn.apache.org/viewvc/?view=rev&rev=1187155
          Files :

          • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java
          • /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
          Show
          Hudson added a comment - Integrated in flume-728 #50 (See https://builds.apache.org/job/flume-728/50/ ) FLUME-728 . Added a multi-threaded test to exercise JDBC channel. arvind : http://svn.apache.org/viewvc/?view=rev&rev=1187155 Files : /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/MockEventUtils.java /incubator/flume/branches/flume-728/flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
          Hide
          E. Sammer added a comment -

          I'm declaring this resolved having hit a reasonable milestone. At this point, specific bugs, feature requests, and tasks should be filed against the NG branch.

          Show
          E. Sammer added a comment - I'm declaring this resolved having hit a reasonable milestone. At this point, specific bugs, feature requests, and tasks should be filed against the NG branch.
          E. Sammer made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          E. Sammer made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              E. Sammer
              Reporter:
              E. Sammer
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development