Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2917

Provide netcat UDP source as alternative to TCP

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.8.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently Flume provides a Netcat TCP source, however Netcat is often used with UDP. There is an implementation of a UDP client in the SyslogUDP source, this request takes this implementation and strips out the Syslog parts, thus forming a Netcat UDP source - where each datagram is recorded as a Flume event.

      The implementation is provided for this at https://github.com/tmgstevens/FlumeNetcatUDPSource and also provided as an attached patch for inclusion.

      N.B. Unit tests are provided for this.

      1. FLUME-2917.patch
        12 kB
        Tristan Stevens
      2. FLUME-2917-7.patch
        17 kB
        Tristan Stevens

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #255 (See https://builds.apache.org/job/Flume-trunk-hbase-1/255/)
          FLUME-2917. Provide netcat UDP source as alternative to TCP (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef54f9fd6372143adc8c6a66f174feba381fec95)

          • (add) flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatUdpSource.java
          • (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java
          • (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java
          • (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java
          • (add) flume-ng-core/src/main/java/org/apache/flume/source/NetcatUdpSource.java
          • (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #255 (See https://builds.apache.org/job/Flume-trunk-hbase-1/255/ ) FLUME-2917 . Provide netcat UDP source as alternative to TCP (bessbd: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ef54f9fd6372143adc8c6a66f174feba381fec95 ) (add) flume-ng-core/src/test/java/org/apache/flume/source/TestNetcatUdpSource.java (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java (edit) flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java (edit) flume-ng-core/src/test/java/org/apache/flume/source/TestDefaultSourceFactory.java (add) flume-ng-core/src/main/java/org/apache/flume/source/NetcatUdpSource.java (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Lior Zeno, Chris Horrocks: thank you for the review!

          Show
          bessbd Bessenyei Balázs Donát added a comment - Lior Zeno , Chris Horrocks : thank you for the review!
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Tristan Stevens: thank you for the patch!

          Show
          bessbd Bessenyei Balázs Donát added a comment - Tristan Stevens : thank you for the patch!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ef54f9fd6372143adc8c6a66f174feba381fec95 in flume's branch refs/heads/trunk from Tristan Stevens
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ef54f9f ]

          FLUME-2917. Provide netcat UDP source as alternative to TCP

          This patch adds a netcat UDP source.

          Reviewers: Lior Zeno, Chris Horrocks, Bessenyei Balázs Donát

          (Tristan Stevens via Bessenyei Balázs Donát)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ef54f9fd6372143adc8c6a66f174feba381fec95 in flume's branch refs/heads/trunk from Tristan Stevens [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ef54f9f ] FLUME-2917 . Provide netcat UDP source as alternative to TCP This patch adds a netcat UDP source. Reviewers: Lior Zeno, Chris Horrocks, Bessenyei Balázs Donát (Tristan Stevens via Bessenyei Balázs Donát)
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          LGTM, +1

          I'm about to commit this.

          (Re-checked, still seems to be working fine.)

          Show
          bessbd Bessenyei Balázs Donát added a comment - LGTM, +1 I'm about to commit this. (Re-checked, still seems to be working fine.)
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Tristan Stevens: I'd be happy to!

          I'll do a final check in the next few days.

          Also, I'll leave some time for others to review and add comments if they have any.

          Show
          bessbd Bessenyei Balázs Donát added a comment - Tristan Stevens : I'd be happy to! I'll do a final check in the next few days. Also, I'll leave some time for others to review and add comments if they have any.
          Hide
          tmgstev Tristan Stevens added a comment -

          Final patch from review board, rebased against trunk

          Show
          tmgstev Tristan Stevens added a comment - Final patch from review board, rebased against trunk
          Hide
          tmgstev Tristan Stevens added a comment -

          Bessenyei Balázs Donát I've fixed the final defect (sorry about the delay). Are you able to commit this?

          Show
          tmgstev Tristan Stevens added a comment - Bessenyei Balázs Donát I've fixed the final defect (sorry about the delay). Are you able to commit this?
          Hide
          ChrisHorrocks Chris Horrocks added a comment -

          Tristan, is there any help you need with this?

          The existing SyslogUDP source has timestamp handling which prevents it's use with non RFC-3164 formatted logs, and this source would be highly beneficial in getting around that constraint.

          Show
          ChrisHorrocks Chris Horrocks added a comment - Tristan, is there any help you need with this? The existing SyslogUDP source has timestamp handling which prevents it's use with non RFC-3164 formatted logs, and this source would be highly beneficial in getting around that constraint.
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Tristan Stevens: I have left one final review comment.
          I can make the change if you agree before committing your patch.

          Show
          bessbd Bessenyei Balázs Donát added a comment - Tristan Stevens : I have left one final review comment. I can make the change if you agree before committing your patch.
          Hide
          CHorrocks Chris Horrocks added a comment -

          We've been testing this for some time in our lab alongside its TCP equivalent with no issues.

          Show
          CHorrocks Chris Horrocks added a comment - We've been testing this for some time in our lab alongside its TCP equivalent with no issues.
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          With the latest changes, the patch looks good to me.

          I'll leave some time for others to review and add comments if they have any.

          Show
          bessbd Bessenyei Balázs Donát added a comment - With the latest changes, the patch looks good to me. I'll leave some time for others to review and add comments if they have any.
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Hi Tristan Stevens,

          I have left some review comments on the review board entry.

          Please, let me know what you think.

          Show
          bessbd Bessenyei Balázs Donát added a comment - Hi Tristan Stevens , I have left some review comments on the review board entry. Please, let me know what you think.
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Tristan Stevens: sorry, I've somehow missed your comment. I'll review the patch as soon as possible.

          If anyone else has comments, please let us know.

          Show
          bessbd Bessenyei Balázs Donát added a comment - Tristan Stevens : sorry, I've somehow missed your comment. I'll review the patch as soon as possible. If anyone else has comments, please let us know.
          Hide
          tmgstev Tristan Stevens added a comment -

          Bessenyei Balázs Donát Lior Zeno Thanks for the review comments. Is there anything else needed for this from me?

          Show
          tmgstev Tristan Stevens added a comment - Bessenyei Balázs Donát Lior Zeno Thanks for the review comments. Is there anything else needed for this from me?
          Hide
          bessbd Bessenyei Balázs Donát added a comment - - edited

          Tristan Stevens: thank you for the reviewboard request

          I have left some comments. Please, let me know what you think

          Show
          bessbd Bessenyei Balázs Donát added a comment - - edited Tristan Stevens : thank you for the reviewboard request I have left some comments. Please, let me know what you think
          Hide
          tmgstev Tristan Stevens added a comment -

          Just looking back at how I did this - it was actually inspired by the SyslogUDP source - if anything it should be a superclass of that - although that's probably more re-engineering than is necessary (unless you disagree of course!)

          Given how dissimilar this is vs. the TCP version of Netcat source, I'd propose to keep them separate. I'll create a review request now.

          Show
          tmgstev Tristan Stevens added a comment - Just looking back at how I did this - it was actually inspired by the SyslogUDP source - if anything it should be a superclass of that - although that's probably more re-engineering than is necessary (unless you disagree of course!) Given how dissimilar this is vs. the TCP version of Netcat source, I'd propose to keep them separate. I'll create a review request now.
          Hide
          bessbd Bessenyei Balázs Donát added a comment -

          Tristan Stevens: thank you for the patch!

          I think, this could be a useful change

          I just don't know if it's easier to maintain the feature in a separate module or we should merge it with Netcat Source and add an extra parameter to decide if it's TCP or UDP? What is your opinion?

          Also, could you please create a review request on https://reviews.apache.org ?

          Show
          bessbd Bessenyei Balázs Donát added a comment - Tristan Stevens : thank you for the patch! I think, this could be a useful change I just don't know if it's easier to maintain the feature in a separate module or we should merge it with Netcat Source and add an extra parameter to decide if it's TCP or UDP? What is your opinion? Also, could you please create a review request on https://reviews.apache.org ?
          Hide
          tmgstev Tristan Stevens added a comment -

          Bessenyei Balázs Donát Do you think this source has value? I'd need to add a section to the docs, but I've seen many cases where UDP would be preferable to TCP.

          Show
          tmgstev Tristan Stevens added a comment - Bessenyei Balázs Donát Do you think this source has value? I'd need to add a section to the docs, but I've seen many cases where UDP would be preferable to TCP.

            People

            • Assignee:
              tmgstev Tristan Stevens
              Reporter:
              tmgstev Tristan Stevens
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development