Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2875

Allow RollingFileSink to specify a file prefix and a file extension.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently the RollingFileSink is hard-wired to use a specific PathManager that creates file based on a timestamp and and incrementing value. User's should have the ability to add a prefix to the file name and to add a file extension to properly identify the type of data in the file. In addition, user's should have the ability to provide their own PathManager implementation to allow them to construct the file names and locations however they desire.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Flume-trunk-hbase-1 #148 (See https://builds.apache.org/job/Flume-trunk-hbase-1/148/)
          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7962ce63bc1cfdfed6b54b8b211e785ccab350f5)

          • flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/RollTimePathManager.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerType.java
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManager.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/DefaultPathManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #148 (See https://builds.apache.org/job/Flume-trunk-hbase-1/148/ ) FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7962ce63bc1cfdfed6b54b8b211e785ccab350f5 ) flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/RollTimePathManager.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerType.java flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManager.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerFactory.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/DefaultPathManager.java
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Committed! Thanks Ralph Goers!

          Show
          hshreedharan Hari Shreedharan added a comment - Committed! Thanks Ralph Goers !
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b03ad71685b1032083145ee0a48ac5e4de8570a3 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b03ad71 ]

          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file extension.

          (Ralph Goers via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit b03ad71685b1032083145ee0a48ac5e4de8570a3 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b03ad71 ] FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file extension. (Ralph Goers via Hari)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 7962ce63bc1cfdfed6b54b8b211e785ccab350f5 in flume's branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7962ce6 ]

          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file extension.

          (Ralph Goers via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 7962ce63bc1cfdfed6b54b8b211e785ccab350f5 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7962ce6 ] FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file extension. (Ralph Goers via Hari)
          Hide
          hshreedharan Hari Shreedharan added a comment -

          +1. Running tests and committing

          Show
          hshreedharan Hari Shreedharan added a comment - +1. Running tests and committing
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Updated patch to allow file name to use current time and date

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Updated patch to allow file name to use current time and date
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Sure - I will take a look at your patch.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Sure - I will take a look at your patch.
          Hide
          praste Pushkar Raste added a comment -

          Ralph Goers - I did some similar work while working patch for FLUME-2856. It has a few other fixes you do be interested in. Please take a look

          Show
          praste Pushkar Raste added a comment - Ralph Goers - I did some similar work while working patch for FLUME-2856 . It has a few other fixes you do be interested in. Please take a look
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Added patch that allows custom PathManagers and allows a file prefix and file extension in the default PathManager.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Added patch that allows custom PathManagers and allows a file prefix and file extension in the default PathManager.

            People

            • Assignee:
              ralph.goers@dslextreme.com Ralph Goers
              Reporter:
              ralph.goers@dslextreme.com Ralph Goers
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development