Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2456

Unable to use FileChannel on Flume 1.5.0.1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: None
    • Component/s: File Channel
    • Labels:
      None

      Description

      I am unable to send logs when I am using a FileChannel in Flume 1.5.0.1. Here is the stack trace that I am seeing:

      java.lang.IllegalStateException: Channel closed [channel=channel]. Due to java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
      at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
      at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
      at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:336)
      at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
      at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
      at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
      at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
      at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
      at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
      at org.apache.flume.channel.file.Log.replay(Log.java:449)
      at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
      at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      ... 1 more

      When I change the agent configuration to use the memory channel, the error goes away, and my logs get transferred successfully.

      Source host configuration broken looks like this:

      agent.channels = channel
      agent.sinks = sink

      agent.channels.channel.type = file
      agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
      agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data

      agent.sinks.sink.channel = channel
      agent.sinks.sink.type = avro
      agent.sinks.sink.hostname = sink_hostname
      agent.sinks.sink.port = 40000

      agent.sources = source1

      agent.sources.source1.type = exec
      agent.sources.source1.command = tail -F /var/log/app.log
      agent.sources.source1.channels = channel

      However, using the same configuration, I am able to get the logs to successfully transfer when I change the configuration to use a memory channel instead:

      By adding this:

      agent.channels.channel.type = memory

      And removing the file channel configuration:

      agent.channels.channel.type = file
      agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
      agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data

        Attachments

          Activity

            People

            • Assignee:
              jrufus Johny Rufus
              Reporter:
              huckphin Chad Rhyner
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: