[FLUME-3264] Null point using hdfs sink when flume startup faster than hdfs

• Type: Bug
• Status: Patch Available
• Priority: Trivial
• Resolution: Unresolved
• Affects Version/s: 1.7.0, 1.8.0
• Component/s: Sinks+Sources

Environment

flume 1.8.0, hdfs 1.9.1.

 

• Release Note: Add null point check at flush(), to get around null point exception in case flume startup faster than hdfs.
• Assignee: Unassigned
• Reporter: wangzheng
• Created: 28/Jul/18 13:30
• Updated: 03/Aug/18 05:07

Description

When flume starts up faster than hdfs, then hdfs sink will report NullPoint, nothing will sink to hdfs. Patch have been tested in my dev env.

 

flume_1 | 2018-07-25 03:49:06,147 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:582)] Caught IOException writing to HDFSWriter (opt). Closing file (hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp) and rethrowing exception.
flume_1 | 2018-07-25 03:49:06,161 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:393)] Closing hdfs://namenode:9000/flume/log00/201807/FlumeData.1532490542456.avro.tmp
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:447)] process failed
flume_1 | java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701)
flume_1 | at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:698)
flume_1 | at java.util.concurrent.FutureTask.run(FutureTask.java:266)
flume_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
flume_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | 2018-07-25 03:49:06,164 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
flume_1 | org.apache.flume.EventDeliveryException: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
flume_1 | at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
flume_1 | at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
flume_1 | at java.lang.Thread.run(Thread.java:748)
flume_1 | Caused by: java.lang.NullPointerException
flume_1 | at org.apache.flume.sink.hdfs.AvroEventSerializer.flush(AvroEventSerializer.java:187)
flume_1 | at org.apache.flume.sink.hdfs.HDFSDataStream.close(HDFSDataStream.java:131)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:327)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$3.call(BucketWriter.java:323)
flume_1 | at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:701) 

Issue Links

• links to: Review Request #68179