Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      Adding a new flume source for generic unicast and multicast UDP.

      Description

      There are existent TCP (NetcatSource) and UDP Syslog (SyslogUDPSource) sources for Flume, but nothing that is able to easily consume plain ol' UDP streams. My use case includes unicast UDP, as well as multicast streams.

      I'm working on coding up a generic UDPSource for the Wikimedia Foundation, and was wondering if this would be useful to the Flume community at large. If so, here are some questions:

      If I made a generic UDPSource, a lot could be DRYed and abstracted out of SyslogUDPSource, with SyslogUDPSource becomfing a subclass. Should I bother doing this? I'd almost rather not, as I won't personally be using SyslogUDPSource, so I'd rather not have to test changes there.

      Should UDPSource code live in flume-ng-sources, or in flume-ng-core/src/main/.../sources?

        Activity

        Hide
        Brock Noland added a comment -

        What in the world?! SyslogUDPSource has a class starting with a lower name character! (syslogHandler)

        It looks like you could take SyslogUDPSource, rename it GenericUDPSource and then have a single method the UDPSource could override:

        protected Event convert(MessageEvent)

        As part of this JIRA I don't think it's a requirement to re-factor SyslogUDPSource. That can be done in a later JIRA.

        Show
        Brock Noland added a comment - What in the world?! SyslogUDPSource has a class starting with a lower name character! (syslogHandler) It looks like you could take SyslogUDPSource, rename it GenericUDPSource and then have a single method the UDPSource could override: protected Event convert(MessageEvent) As part of this JIRA I don't think it's a requirement to re-factor SyslogUDPSource. That can be done in a later JIRA.
        Hide
        Andrew Otto added a comment -

        This is my first flume patch, so it probably needs lots of review and changes. Thanks!

        Show
        Andrew Otto added a comment - This is my first flume patch, so it probably needs lots of review and changes. Thanks!
        Hide
        Andrew Otto added a comment -

        This is the second patch I've uploaded. The difference between this and the previous, is that this does not write the delimiter into the Event when it is read. This also reverts the change to Context. I decided not to implement a getCharacter method, as it was just making things more complicated with no real gain.

        Show
        Andrew Otto added a comment - This is the second patch I've uploaded. The difference between this and the previous, is that this does not write the delimiter into the Event when it is read. This also reverts the change to Context. I decided not to implement a getCharacter method, as it was just making things more complicated with no real gain.

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew Otto
          • Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 336h
              336h
              Remaining:
              Remaining Estimate - 336h
              336h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development