[FLUME-2147] Missing headers cause events to become stuck in channel

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Jonathan Cooper-Ellis
• Created: 06/Aug/13 15:59
• Updated: 22/Aug/16 07:20

Description

If a sink expects a header but does not find it, events will become stuck in the channel and Flume will log NullPointer and EventDelivery exceptions. In a memory channel, this can be fixed by restarting. In a file channel, restarting does not cause events to be removed.

05 Aug 2013 12:21:09,424 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:426)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:200)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:396)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:356)
... 3 more
05 Aug 2013 12:21:09,424 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:422) - process failed
java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:200)
at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:396)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:356)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:662)

I was using RegexExtractorInterceptor to match timestamp for partitioning in with HDFS sink.

Issue Links

• duplicates: FLUME-1264 HDFEventSink throws NPE if event is generated without timestamp in header