Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2056

Allow SpoolDir to pass just the filename that is the source of an event

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently we allow for passing of the absolute path.
      It would be nice to just pass the filename in the event headers and allow for using that on the hdfs sink.

      if (annotateFileName) {
      String filename = currentFile.get().getFile().getAbsolutePath();
      for (Event event : events)

      { event.getHeaders().put(fileNameHeader, filename); }

      }

      1. FLUME-2056.0.patch
        11 kB
        Jeff Lord
      2. FLUME-2056-1.patch
        11 kB
        Jeff Lord

        Issue Links

          Activity

          Hide
          Skiv Ivan added a comment -

          +1, or provide interceptor which can modify headers like regex_extract

          Show
          Skiv Ivan added a comment - +1, or provide interceptor which can modify headers like regex_extract
          Hide
          jlord Jeff Lord added a comment -

          Attached is a patch which adds this functionality to put the basename of a file in the event headers. Reviewboard link to follow shortly.

          Show
          jlord Jeff Lord added a comment - Attached is a patch which adds this functionality to put the basename of a file in the event headers. Reviewboard link to follow shortly.
          Hide
          jlord Jeff Lord added a comment -

          Revised unit test.

          while (source.getSourceCounter().getEventAcceptedCount() < 8)

          { Thread.sleep(10); }
          Show
          jlord Jeff Lord added a comment - Revised unit test. while (source.getSourceCounter().getEventAcceptedCount() < 8) { Thread.sleep(10); }
          Hide
          mpercy Mike Percy added a comment -

          +1 LGTM

          Show
          mpercy Mike Percy added a comment - +1 LGTM
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in flume-trunk #530 (See https://builds.apache.org/job/flume-trunk/530/)
          FLUME-2056. Allow SpoolDir to pass just the filename that is the source of an event (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=72b3b7889f1dcf6736f099b77c878efe891c6585)

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
          • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in flume-trunk #530 (See https://builds.apache.org/job/flume-trunk/530/ ) FLUME-2056 . Allow SpoolDir to pass just the filename that is the source of an event (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=72b3b7889f1dcf6736f099b77c878efe891c6585 ) flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java

            People

            • Assignee:
              jlord Jeff Lord
              Reporter:
              jlord Jeff Lord
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development