[FLUME-3217] Flume creates empty files when HDFS quota has been reached

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Denes Arvay
• Created: 16/Feb/18 14:50
• Updated: 17/Aug/18 10:59

Description

Flume creates empty files when HDFS quota has been reached and leaves them on HDFS. 
The file creation was successful, but as long as the quota did not allow any write, new file was created on every write attempt.

Relevant error message from flume log:
 

2018-02-07 14:59:30,563 WARN org.apache.flume.sink.hdfs.BucketWriter: Caught IOException writing to HDFSWriter (The DiskSpace quota of /data/catssolprn is exceeded: quota = 2199023255552 B = 2 TB but diskspace consumed = 2199217840800 B = 2.00 TB
	at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota(DirectoryWithQuotaFeature.java:149)
	at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:159)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota(FSDirectory.java:2037)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1868)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount(FSDirectory.java:1843)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock(FSDirectory.java:441)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3806)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3394)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:683)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:214)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:495)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2141)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2137)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1840)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2135)
). Closing file (/user/flume/events/FlumeData.1518033562880.log.tmp) and rethrowing exception. 

Config for reproduction:

tier1.sources = source1
tier1.channels = channel1
tier1.sinks    = sink1

tier1.sources.source1.type     = netcat
tier1.sources.source1.bind     = 127.0.0.1
tier1.sources.source1.port     = 9999
tier1.sources.source1.channels = channel1

tier1.channels.channel1.type                = memory

tier1.sinks.sink1.type= hdfs
tier1.sinks.sink1.fileType=DataStream
tier1.sinks.sink1.channel = channel1
tier1.sinks.sink1.hdfs.path = hdfs://nameservice1/user/flume/events

hdfs dfs commands:

sudo -u flume hdfs dfs -mkdir -p /user/flume/events
sudo -u hdfs hdfs dfsadmin -setSpaceQuota 3000 /user/flume/events

Issue Links

• links to: GitHub Pull Request #225