Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2056

Allow SpoolDir to pass just the filename that is the source of an event

        Details

        • Type: New Feature
        • Status: Resolved
        • Priority: Minor
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 1.5.0
        • Component/s: None
        • Labels:
          None

          Description

          Currently we allow for passing of the absolute path.
          It would be nice to just pass the filename in the event headers and allow for using that on the hdfs sink.

          if (annotateFileName) {
          String filename = currentFile.get().getFile().getAbsolutePath();
          for (Event event : events)

          { event.getHeaders().put(fileNameHeader, filename); }

          }

          1. Text File
            FLUME-2056.0.patch
            11 kB
            Jeff Lord
          2. Text File
            FLUME-2056-1.patch
            11 kB
            Jeff Lord

            Issue Links

            links to

            Web Link ReviewBoard

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Skiv Ivan added a comment -

              +1, or provide interceptor which can modify headers like regex_extract

              Show
              Skiv Ivan added a comment - +1, or provide interceptor which can modify headers like regex_extract
              Hide
              Permalink
              jlord Jeff Lord added a comment -

              Attached is a patch which adds this functionality to put the basename of a file in the event headers. Reviewboard link to follow shortly.

              Show
              jlord Jeff Lord added a comment - Attached is a patch which adds this functionality to put the basename of a file in the event headers. Reviewboard link to follow shortly.
              Hide
              Permalink
              jlord Jeff Lord added a comment -

              Revised unit test.

              while (source.getSourceCounter().getEventAcceptedCount() < 8)

              { Thread.sleep(10); }
              Show
              jlord Jeff Lord added a comment - Revised unit test. while (source.getSourceCounter().getEventAcceptedCount() < 8) { Thread.sleep(10); }
              Hide
              Permalink
              mpercy Mike Percy added a comment -

              +1 LGTM

              Show
              mpercy Mike Percy added a comment - +1 LGTM
              Hide
              Permalink
              mpercy Mike Percy added a comment -

              Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!

              Show
              mpercy Mike Percy added a comment - Pushed to trunk and flume-1.5 branches. Thanks for the patch Jeff!
              Hide
              Permalink
              hudson Hudson added a comment -

              FAILURE: Integrated in flume-trunk #530 (See https://builds.apache.org/job/flume-trunk/530/)
              FLUME-2056. Allow SpoolDir to pass just the filename that is the source of an event (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=72b3b7889f1dcf6736f099b77c878efe891c6585)

              • flume-ng-doc/sphinx/FlumeUserGuide.rst
              • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
              • flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
              • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
              • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
              Show
              hudson Hudson added a comment - FAILURE: Integrated in flume-trunk #530 (See https://builds.apache.org/job/flume-trunk/530/ ) FLUME-2056 . Allow SpoolDir to pass just the filename that is the source of an event (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=72b3b7889f1dcf6736f099b77c878efe891c6585 ) flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java

                People

                • Assignee:
                  jlord Jeff Lord
                  Reporter:
                  jlord Jeff Lord
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development