[FLUME-2456] Unable to use FileChannel on Flume 1.5.0.1

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.0.1
• Component/s: File Channel
• Assignee: Johny Rufus
• Reporter: Chad Rhyner
• Created: 05/Sep/14 19:02
• Updated: 01/Mar/15 13:41

Description

I am unable to send logs when I am using a FileChannel in Flume 1.5.0.1. Here is the stack trace that I am seeing:

java.lang.IllegalStateException: Channel closed [channel=channel]. Due to java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:352)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:336)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
at org.apache.flume.channel.file.Log.replay(Log.java:449)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more

When I change the agent configuration to use the memory channel, the error goes away, and my logs get transferred successfully.

Source host configuration broken looks like this:

agent.channels = channel
agent.sinks = sink

agent.channels.channel.type = file
agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data

agent.sinks.sink.channel = channel
agent.sinks.sink.type = avro
agent.sinks.sink.hostname = sink_hostname
agent.sinks.sink.port = 40000

agent.sources = source1

agent.sources.source1.type = exec
agent.sources.source1.command = tail -F /var/log/app.log
agent.sources.source1.channels = channel

However, using the same configuration, I am able to get the logs to successfully transfer when I change the configuration to use a memory channel instead:

By adding this:

agent.channels.channel.type = memory

And removing the file channel configuration:

agent.channels.channel.type = file
agent.channels.channel.checkpointDir = /media/ephemeral0/file-channel-test/checkpoint
agent.channels.channel.dataDirs = /media/ephemeral0/file-channel-test/data