Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2497

TCP and UDP syslog sources parsing the timestamp incorrectly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods.

        Activity

        Hide
        jrufus Johny Rufus added a comment -

        This happens in the case where the timestamp is of the form "2014-10-03T17:20:01.222357", where the microseconds is present
        The prime reason being the inability of SimpleDateFormat used by SyslogUtils to parse microseconds. Instead of considering 222357 as microseconds, it is treated as milliseconds.
        In this scenario, we need to specify the value in milliseconds (by discarding the digits after the first three digits). This behavior is in consistent with Joda DateTimeFormatter used by MultiportSyslogTCPSource (SyslogParser).
        Attached a patch that does the above conversion.

        Show
        jrufus Johny Rufus added a comment - This happens in the case where the timestamp is of the form "2014-10-03T17:20:01.222357", where the microseconds is present The prime reason being the inability of SimpleDateFormat used by SyslogUtils to parse microseconds. Instead of considering 222357 as microseconds, it is treated as milliseconds. In this scenario, we need to specify the value in milliseconds (by discarding the digits after the first three digits). This behavior is in consistent with Joda DateTimeFormatter used by MultiportSyslogTCPSource (SyslogParser). Attached a patch that does the above conversion.
        Hide
        jrufus Johny Rufus added a comment -

        Reference to the javadoc for SimpleDateFormat, http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
        which specifies that 'S' corresponds to milliseconds and the standard format that matches the above pattern for parsing is
        yyyy-MM-dd'T'HH:mm:ss.S and in which case S = 222357 is taken as milliseconds

        Show
        jrufus Johny Rufus added a comment - Reference to the javadoc for SimpleDateFormat, http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html which specifies that 'S' corresponds to milliseconds and the standard format that matches the above pattern for parsing is yyyy-MM-dd'T'HH:mm:ss.S and in which case S = 222357 is taken as milliseconds
        Hide
        hshreedharan Hari Shreedharan added a comment -

        Mike Percy - Could you please take a look? I am not too familiar with this code.

        Show
        hshreedharan Hari Shreedharan added a comment - Mike Percy - Could you please take a look? I am not too familiar with this code.
        Hide
        mpercy Mike Percy added a comment -

        +1 lgtm. I have a couple minor nits (add a couple comments, formatting) that I'll just address on commit.

        Show
        mpercy Mike Percy added a comment - +1 lgtm. I have a couple minor nits (add a couple comments, formatting) that I'll just address on commit.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 534fe268d1d0ad197f6e4a867ab0ae0155d1a927 in flume's branch refs/heads/trunk from Mike Percy
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=534fe26 ]

        FLUME-2497. Support fractional seconds in Syslog timestamps

        This fixes a bug in the SyslogTcpSource and SyslogUdpSource where
        fractional timestamps fail to parse.

        (Johny Rufus via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 534fe268d1d0ad197f6e4a867ab0ae0155d1a927 in flume's branch refs/heads/trunk from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=534fe26 ] FLUME-2497 . Support fractional seconds in Syslog timestamps This fixes a bug in the SyslogTcpSource and SyslogUdpSource where fractional timestamps fail to parse. (Johny Rufus via Mike Percy)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit dcc1ce066853a9594508f21a6bd0d7382a4c240e in flume's branch refs/heads/flume-1.6 from Mike Percy
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dcc1ce0 ]

        FLUME-2497. Support fractional seconds in Syslog timestamps

        This fixes a bug in the SyslogTcpSource and SyslogUdpSource where
        fractional timestamps fail to parse.

        (Johny Rufus via Mike Percy)

        Show
        jira-bot ASF subversion and git services added a comment - Commit dcc1ce066853a9594508f21a6bd0d7382a4c240e in flume's branch refs/heads/flume-1.6 from Mike Percy [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dcc1ce0 ] FLUME-2497 . Support fractional seconds in Syslog timestamps This fixes a bug in the SyslogTcpSource and SyslogUdpSource where fractional timestamps fail to parse. (Johny Rufus via Mike Percy)
        Hide
        mpercy Mike Percy added a comment -

        Pushed to the trunk and flume-1.6 branches. Thanks for the patch Rufus!

        Show
        mpercy Mike Percy added a comment - Pushed to the trunk and flume-1.6 branches. Thanks for the patch Rufus!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in flume-trunk #691 (See https://builds.apache.org/job/flume-trunk/691/)
        FLUME-2497. Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=534fe268d1d0ad197f6e4a867ab0ae0155d1a927)

        • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
        • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #691 (See https://builds.apache.org/job/flume-trunk/691/ ) FLUME-2497 . Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=534fe268d1d0ad197f6e4a867ab0ae0155d1a927 ) flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #48 (See https://builds.apache.org/job/Flume-trunk-hbase-98/48/)
        FLUME-2497. Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=534fe268d1d0ad197f6e4a867ab0ae0155d1a927)

        • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
        • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #48 (See https://builds.apache.org/job/Flume-trunk-hbase-98/48/ ) FLUME-2497 . Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=534fe268d1d0ad197f6e4a867ab0ae0155d1a927 ) flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java

          People

          • Assignee:
            jrufus Johny Rufus
            Reporter:
            jrufus Johny Rufus
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development