Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2982

Add localhost escape sequence to HDFS sink

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0
    • Component/s: None
    • Labels:
      None

      Description

      It would be useful to be able to just use %localhost% escape sequence instead of having to pass in a header / use the host interceptor.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #203 (See https://builds.apache.org/job/Flume-trunk-hbase-1/203/)
          FLUME-2982. Add localhost escape sequence to HDFS sink (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a0a50849d5ad33e6e6903316a3bd3dbba8547843)

          • (edit) flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java
          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          • (edit) flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #203 (See https://builds.apache.org/job/Flume-trunk-hbase-1/203/ ) FLUME-2982 . Add localhost escape sequence to HDFS sink (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=a0a50849d5ad33e6e6903316a3bd3dbba8547843 ) (edit) flume-ng-core/src/main/java/org/apache/flume/formatter/output/BucketPath.java (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst (edit) flume-ng-core/src/test/java/org/apache/flume/formatter/output/TestBucketPath.java
          Hide
          mpercy Mike Percy added a comment -

          Pushed to trunk. Thanks for the patch Jeff!

          Show
          mpercy Mike Percy added a comment - Pushed to trunk. Thanks for the patch Jeff!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit a0a50849d5ad33e6e6903316a3bd3dbba8547843 in flume's branch refs/heads/trunk from Jeff Holoman
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a0a5084 ]

          FLUME-2982. Add localhost escape sequence to HDFS sink

          It would be useful to be able to just use %[localhost] escape sequence
          instead of having to pass in a header or use the host interceptor.

          Part of the problem of using an interceptor is the case where Flume only
          consists of a channel and a sink (think KafkaChannel). There is support
          in Flume for a sink-side interceptor at the time of this writing.

          Reviewers: Grant Henke, Mike Percy

          (Jeff Holoman via Mike Percy)

          Show
          jira-bot ASF subversion and git services added a comment - Commit a0a50849d5ad33e6e6903316a3bd3dbba8547843 in flume's branch refs/heads/trunk from Jeff Holoman [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=a0a5084 ] FLUME-2982 . Add localhost escape sequence to HDFS sink It would be useful to be able to just use % [localhost] escape sequence instead of having to pass in a header or use the host interceptor. Part of the problem of using an interceptor is the case where Flume only consists of a channel and a sink (think KafkaChannel). There is support in Flume for a sink-side interceptor at the time of this writing. Reviewers: Grant Henke, Mike Percy (Jeff Holoman via Mike Percy)
          Hide
          mpercy Mike Percy added a comment -

          +1. I am about to commit this.

          Show
          mpercy Mike Percy added a comment - +1. I am about to commit this.
          Hide
          mpercy Mike Percy added a comment -

          Hey Jeff, thanks for testing. If it seems to work fine then I'm ok with it

          Show
          mpercy Mike Percy added a comment - Hey Jeff, thanks for testing. If it seems to work fine then I'm ok with it
          Hide
          jholoman Jeff Holoman added a comment -

          Thanks @mpercy I took a look at this and tested on a few different hosts and the InetAddress.getLocalHost() approach seemed to work ok. This is also how we do it now for the existing host intereceptor so I think we are ok, per the comments in the RB.

          Show
          jholoman Jeff Holoman added a comment - Thanks @mpercy I took a look at this and tested on a few different hosts and the InetAddress.getLocalHost() approach seemed to work ok. This is also how we do it now for the existing host intereceptor so I think we are ok, per the comments in the RB.
          Hide
          mpercy Mike Percy added a comment -

          Sounds like a useful feature. Consider using syntax compatible with the existing stuff, but not to be confused with the header syntax.

          Perhaps %(localhost) or something like that.

          By the way, getting the hostname from the local machine isn't as easy as you might expect in Java. See the comments on this SO thread: http://stackoverflow.com/questions/7348711/recommended-way-to-get-hostname-in-java

          Possibly the most reliable way is to shell out and execute hostname – this sounds like it should be portable since you should end up with /bin/hostname on *Nix and HOSTNAME.EXE on Windows. See http://stackoverflow.com/a/28043703

          This may be worth writing a utility function for that caches the result, possibly invalidating that cache when the sink is reconfigured or something like that.

          Show
          mpercy Mike Percy added a comment - Sounds like a useful feature. Consider using syntax compatible with the existing stuff, but not to be confused with the header syntax. Perhaps %(localhost) or something like that. By the way, getting the hostname from the local machine isn't as easy as you might expect in Java. See the comments on this SO thread: http://stackoverflow.com/questions/7348711/recommended-way-to-get-hostname-in-java Possibly the most reliable way is to shell out and execute hostname – this sounds like it should be portable since you should end up with /bin/hostname on *Nix and HOSTNAME.EXE on Windows. See http://stackoverflow.com/a/28043703 This may be worth writing a utility function for that caches the result, possibly invalidating that cache when the sink is reconfigured or something like that.

            People

            • Assignee:
              jholoman Jeff Holoman
              Reporter:
              jholoman Jeff Holoman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development