Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2798

Malformed Syslog messages can lead to OutOfMemoryException

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.5.0, 1.6.0
    • Fix Version/s: 1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      It's possible for a client submitting syslog data which is malformed in various ways to convince SyslogUtils.extractEvent to continually fill the ByteArrayOutputStream it uses to collect the event until the agent runs out of memory. Since the OOM condition affects the whole agent, it's possible that a client sending such data (due to accident or malicious intent) to disable the agent, as long as it remains connected.

      Note that this is probably only possible using SyslogTcpSource although the fix touches common code in SyslogUtils.java.

      The issue can happen in two ways:

      Scenario 1: Send a message like this:

      <> some more stuff here

      This causes a NumberFormatException:

      Sep 11, 2015 2:27:07 AM org.jboss.netty.channel.SimpleChannelHandler
      WARNING: EXCEPTION, please implement org.apache.flume.source.SyslogTcpSource$syslogTcpHandler.exceptionCaught() for proper handling.
      java.lang.NumberFormatException: For input string: ""
      	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      	at java.lang.Integer.parseInt(Integer.java:504)
      	at java.lang.Integer.parseInt(Integer.java:527)
      	at org.apache.flume.source.SyslogUtils.buildEvent(SyslogUtils.java:198)
      	at org.apache.flume.source.SyslogUtils.extractEvent(SyslogUtils.java:344)
      	at org.apache.flume.source.SyslogTcpSource$syslogTcpHandler.messageReceived(SyslogTcpSource.java:76)
      	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
      	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
      	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      This exception does not get handled, and it happens before reset() can be called. The result is that the state machine in SyslogUtils gets stuck in the DATA state, and all subsequent data just gets appended to the baos, while the above exception streams to the log. Eventually the agent runs out of memory.

      Scenario 2: Send some data like this:

      <123...........

      No length checking is done in the PRIO state so you could potentially fill the agent memory this way too.

      I'm attaching a patch which handles both of these issues and adds more exception handling to buildEvent to make sure that reset() is called in future unforeseen situations.

      Thanks also to Roshan Naik for helping to make this patch better.

        Activity

        Hide
        roshan_naik Roshan Naik added a comment -

        This patch looks good to me.
        +1

        Show
        roshan_naik Roshan Naik added a comment - This patch looks good to me. +1
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c8eb221c6bbeb24a6e5073eb89e69e7cd2a10366 in flume's branch refs/heads/trunk from Roshan Naik
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c8eb221 ]

        FLUME-2798. Malformed Syslog messages can lead to OutOfMemoryException

        (Phil D'Amore via Roshan Naik)

        Show
        jira-bot ASF subversion and git services added a comment - Commit c8eb221c6bbeb24a6e5073eb89e69e7cd2a10366 in flume's branch refs/heads/trunk from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c8eb221 ] FLUME-2798 . Malformed Syslog messages can lead to OutOfMemoryException (Phil D'Amore via Roshan Naik)
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7dbe6fef6226d3fc2d8ada711c583b29ede3cfbd in flume's branch refs/heads/flume-1.7 from Roshan Naik
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7dbe6fe ]

        FLUME-2798. Malformed Syslog messages can lead to OutOfMemoryException

        (Phil D'Amore via Roshan Naik)

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7dbe6fef6226d3fc2d8ada711c583b29ede3cfbd in flume's branch refs/heads/flume-1.7 from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=7dbe6fe ] FLUME-2798 . Malformed Syslog messages can lead to OutOfMemoryException (Phil D'Amore via Roshan Naik)
        Hide
        roshan_naik Roshan Naik added a comment -

        Its committed.

        Thanks very much for the patch Phil D'Amore

        Show
        roshan_naik Roshan Naik added a comment - Its committed. Thanks very much for the patch Phil D'Amore
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Flume-trunk-hbase-1 #128 (See https://builds.apache.org/job/Flume-trunk-hbase-1/128/)
        FLUME-2798. Malformed Syslog messages can lead to OutOfMemoryException (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c8eb221c6bbeb24a6e5073eb89e69e7cd2a10366)

        • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java
        • flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java
        • flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java
        • flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Flume-trunk-hbase-1 #128 (See https://builds.apache.org/job/Flume-trunk-hbase-1/128/ ) FLUME-2798 . Malformed Syslog messages can lead to OutOfMemoryException (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=c8eb221c6bbeb24a6e5073eb89e69e7cd2a10366 ) flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java

          People

          • Assignee:
            tweek Phil D'Amore
            Reporter:
            tweek Phil D'Amore
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development