Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2403

FileChannel causes OOME for large messages.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.5.0
    • Fix Version/s: None
    • Component/s: Channel
    • Labels:
      None

      Description

      The spillable memory channel will fail rather badly on large messages.

      Error while writing to required channel: FileChannel es1 { dataDirs: [/var/lib/flume/data] }
      java.lang.OutOfMemoryError: Java heap space
              at java.util.Arrays.copyOf(Arrays.java:2271)
              at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
              at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
              at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
              at com.google.protobuf.CodedOutputStream.writeRawBytes(CodedOutputStream.java:984)
              at com.google.protobuf.CodedOutputStream.writeRawBytes(CodedOutputStream.java:905)
              at com.google.protobuf.CodedOutputStream.writeBytesNoTag(CodedOutputStream.java:386)
              at com.google.protobuf.CodedOutputStream.writeBytes(CodedOutputStream.java:229)
              at org.apache.flume.channel.file.proto.ProtosFactory$FlumeEvent.writeTo(ProtosFactory.java:6259)
              at com.google.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:380)
              at com.google.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:222)
              at org.apache.flume.channel.file.proto.ProtosFactory$Put.writeTo(ProtosFactory.java:4112)
              at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:90)
              at org.apache.flume.channel.file.Put.writeProtos(Put.java:93)
              at org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
              at org.apache.flume.channel.file.Log.put(Log.java:611)
              at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:458)
              at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
              at org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.commitPutsToOverflow(SpillableMemoryChannel.java:490)
              at org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.putCommit(SpillableMemoryChannel.java:480)
              at org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doCommit(SpillableMemoryChannel.java:401)
              at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
              at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:267)
              at org.apache.flume.source.rabbitmq.RabbitMQSource.process(RabbitMQSource.java:162)
              at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
              at java.lang.Thread.run(Thread.java:744)
      

      My config:

      agent.channels.es1.type = SPILLABLEMEMORY
      agent.channels.es1.memoryCapacity = 10000
      agent.channels.es1.overflowCapacity = 1000000
      agent.channels.es1.byteCapacity = 800000
      agent.channels.es1.checkpointDir = /var/lib/flume/checkpoint
      agent.channels.es1.dataDirs = /var/lib/flume/data
      

      I haven't looked at the code but I have some concerns like why a ByteArrayOutputStream is being used instead of some other buffered stream directly to the file system? Perhaps its because of the transactional nature but I'm pretty sure you can write to the filesystem and rollback as Kafka and modern databases do this with fsync.

      One could argue that I should just raise the max heap but this message is coming from a RabbitMQ which had no issue holding on to the message (I believe the message is like 500K).

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              agentgt Adam Gent
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: