[FLUME-3147] LeaseExpiredException while storing files into HDFS

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Sinks+Sources
• Flags: Patch
• Assignee: Unassigned
• Reporter: Kallam Reddy
• Created: 15/Aug/17 13:39
• Updated: 15/Aug/17 13:39

Description

While storing events from Kafka into HDFS as avro containers we are getting LeaseExpiredException exception, exception trace is given below.

It is because of multiple agents are trying to writer files into a particular folder and the counter being generated to use in the file name is matched for couple of agents.

Problematic code is at: org.apache.flume.sink.hdfs.BucketWriter.java
Issue is with getting the counter in org.apache.flume.sink.hdfs.BucketWriter.java, which might be atomic for one agent but could not be atomic with multiple agents in distributed environment.
long counter = fileExtensionCounter.incrementAndGet();
String fullFileName = fileName + "." + counter;

2017-08-01 19:06:00.634 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.AbstractHDFS
Writer.isUnderReplicated:95) - Unexpected error while checking replication factor
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:161)
at org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:81)
at com.omnitracs.otda.dte.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:631)
at com.omnitracs.otda.dte.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:565)
at com.omnitracs.otda.dte.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:430)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /abc/xyz.1501545600001.avro.tmp (inode 9165696): File does not exist.