Flume
  1. Flume
  2. FLUME-2875

Allow RollingFileSink to specify a file prefix and a file extension.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.6.0
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently the RollingFileSink is hard-wired to use a specific PathManager that creates file based on a timestamp and and incrementing value. User's should have the ability to add a prefix to the file name and to add a file extension to properly identify the type of data in the file. In addition, user's should have the ability to provide their own PathManager implementation to allow them to construct the file names and locations however they desire.

        Issue Links

          Activity

          Hide
          Ralph Goers added a comment -

          Added patch that allows custom PathManagers and allows a file prefix and file extension in the default PathManager.

          Show
          Ralph Goers added a comment - Added patch that allows custom PathManagers and allows a file prefix and file extension in the default PathManager.
          Hide
          Pushkar Raste added a comment -

          Ralph Goers - I did some similar work while working patch for FLUME-2856. It has a few other fixes you do be interested in. Please take a look

          Show
          Pushkar Raste added a comment - Ralph Goers - I did some similar work while working patch for FLUME-2856 . It has a few other fixes you do be interested in. Please take a look
          Hide
          Ralph Goers added a comment -

          Sure - I will take a look at your patch.

          Show
          Ralph Goers added a comment - Sure - I will take a look at your patch.
          Hide
          Ralph Goers added a comment -

          Updated patch to allow file name to use current time and date

          Show
          Ralph Goers added a comment - Updated patch to allow file name to use current time and date
          Hide
          Hari Shreedharan added a comment -

          +1. Running tests and committing

          Show
          Hari Shreedharan added a comment - +1. Running tests and committing
          Hide
          ASF subversion and git services added a comment -

          Commit 7962ce63bc1cfdfed6b54b8b211e785ccab350f5 in flume's branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7962ce6 ]

          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file extension.

          (Ralph Goers via Hari)

          Show
          ASF subversion and git services added a comment - Commit 7962ce63bc1cfdfed6b54b8b211e785ccab350f5 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7962ce6 ] FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file extension. (Ralph Goers via Hari)
          Hide
          ASF subversion and git services added a comment -

          Commit b03ad71685b1032083145ee0a48ac5e4de8570a3 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b03ad71 ]

          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file extension.

          (Ralph Goers via Hari)

          Show
          ASF subversion and git services added a comment - Commit b03ad71685b1032083145ee0a48ac5e4de8570a3 in flume's branch refs/heads/flume-1.7 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=b03ad71 ] FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file extension. (Ralph Goers via Hari)
          Hide
          Hari Shreedharan added a comment -

          Committed! Thanks Ralph Goers!

          Show
          Hari Shreedharan added a comment - Committed! Thanks Ralph Goers !
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Flume-trunk-hbase-1 #148 (See https://builds.apache.org/job/Flume-trunk-hbase-1/148/)
          FLUME-2875. Allow RollingFileSink to specify a file prefix and a file (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7962ce63bc1cfdfed6b54b8b211e785ccab350f5)

          • flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/RollTimePathManager.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerType.java
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManager.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerFactory.java
          • flume-ng-core/src/main/java/org/apache/flume/formatter/output/DefaultPathManager.java
          Show
          Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #148 (See https://builds.apache.org/job/Flume-trunk-hbase-1/148/ ) FLUME-2875 . Allow RollingFileSink to specify a file prefix and a file (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=7962ce63bc1cfdfed6b54b8b211e785ccab350f5 ) flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/RollTimePathManager.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerType.java flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManager.java flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/formatter/output/PathManagerFactory.java flume-ng-core/src/main/java/org/apache/flume/formatter/output/DefaultPathManager.java

            People

            • Assignee:
              Ralph Goers
              Reporter:
              Ralph Goers
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development