Flume
  1. Flume
  2. FLUME-1277

Error parsing Syslog rfc 3164 messages with null values

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.1.0, v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The SyslogUtils class doesn't properly parse rfc 3164 style messages containing a null (hyphen) value. e.g.,
      <10>Apr 1 13:14:04 ubuntu-11.cloudera.com - rest_of_message

      It tries to parse it as a 5424 style message, skips over the date information, and interprets the first hyphen as a null timestamp. Part of the problem is the use of a Scanner and regex. This skips over a properly formatted 3164 style message until it finds anything that matches the 5424 regex, including a hyphen.

      1. FLUME-1277-1a.patch
        7 kB
        Mike Percy
      2. FLUME-1277-1.patch
        7 kB
        Brock Noland
      3. FLUME-1277-fix.patch
        5 kB
        Brent Halsey
      4. FLUME-1277-test.patch
        2 kB
        Brent Halsey

        Issue Links

          Activity

          Hide
          Brent Halsey added a comment -

          Unit test for this issue

          Show
          Brent Halsey added a comment - Unit test for this issue
          Hide
          Brent Halsey added a comment -

          Attaching test case for issue and a fix.

          The fix does a full regex match against the message instead of using Scanner. This revealed a bug in the regex and how it handles the <version>. Also allows us to compile the regex pattern just once on init.

          Spelling correction for SyslogFormatter.

          Show
          Brent Halsey added a comment - Attaching test case for issue and a fix. The fix does a full regex match against the message instead of using Scanner. This revealed a bug in the regex and how it handles the <version>. Also allows us to compile the regex pattern just once on init. Spelling correction for SyslogFormatter.
          Hide
          Hari Shreedharan added a comment -

          Brent,

          Thanks for the patch. Could you please post this on reviewboard(https://reviews.apache.org - add yourself to the Flume group and post the review to that group too) so that committers can review it easily?

          Also: I am assigning this bug to you, since you already worked on it. If you think this patch does not fix the issue and if you no longer want to work on it, please let me know.

          Show
          Hari Shreedharan added a comment - Brent, Thanks for the patch. Could you please post this on reviewboard( https://reviews.apache.org - add yourself to the Flume group and post the review to that group too) so that committers can review it easily? Also: I am assigning this bug to you, since you already worked on it. If you think this patch does not fix the issue and if you no longer want to work on it, please let me know.
          Hide
          Josh West added a comment -

          Will this bugfix be applied to a new 1.2.x release? I'm running into it with syslog collection. The rest_of_message seems to also become a new Flume Event... causing other issues, as this new event doesn't have headers like "host" set.

          Show
          Josh West added a comment - Will this bugfix be applied to a new 1.2.x release? I'm running into it with syslog collection. The rest_of_message seems to also become a new Flume Event... causing other issues, as this new event doesn't have headers like "host" set.
          Hide
          Brock Noland added a comment -

          Brent, can you address Hari's comments or if you no longer want to take this up, say so?

          Josh, Yes, I'd like to get this into the 1.3 release. I will mark it as such. If we don't here from Brent we'll take up the patch.

          Show
          Brock Noland added a comment - Brent, can you address Hari's comments or if you no longer want to take this up, say so? Josh, Yes, I'd like to get this into the 1.3 release. I will mark it as such. If we don't here from Brent we'll take up the patch.
          Hide
          Brock Noland added a comment -

          Rebased the patch, merged it into a single file, and put on RB.

          Show
          Brock Noland added a comment - Rebased the patch, merged it into a single file, and put on RB.
          Hide
          Mike Percy added a comment -

          Attaching latest patch from RB

          Show
          Mike Percy added a comment - Attaching latest patch from RB
          Hide
          Mike Percy added a comment -

          Patch committed. Thanks Brent and Brock!

          Rev: 4c6faee565518d154672210c6418bdad0bc42e85

          Show
          Mike Percy added a comment - Patch committed. Thanks Brent and Brock! Rev: 4c6faee565518d154672210c6418bdad0bc42e85
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #324 (See https://builds.apache.org/job/flume-trunk/324/)
          FLUME-1277. Error parsing Syslog rfc 3164 messages with null values. (Revision 4c6faee565518d154672210c6418bdad0bc42e85)

          Result = UNSTABLE
          mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4c6faee565518d154672210c6418bdad0bc42e85
          Files :

          • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
          • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
          Show
          Hudson added a comment - Integrated in flume-trunk #324 (See https://builds.apache.org/job/flume-trunk/324/ ) FLUME-1277 . Error parsing Syslog rfc 3164 messages with null values. (Revision 4c6faee565518d154672210c6418bdad0bc42e85) Result = UNSTABLE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4c6faee565518d154672210c6418bdad0bc42e85 Files : flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brent Halsey
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development