[FLUME-1370] HDFSEventSink - file name collision in bucket path

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0
• Component/s: Sinks+Sources
• Labels: newbie

Environment

Linux, Java 1.6.0.24, hadoop-0.20.205, flume-1.2.0

• Assignee: Unassigned
• Reporter: Mubarak Seyed
• Created: 13/Jul/12 19:47
• Updated: 13/Sep/12 15:08

Description

It appears from test that two HDFS sinks (from different agent/machines) are trying to create same file name

2012-07-12 22:18:51,820 WARN org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.startFile: failed to create file /logs/event1/07122012/22/2/Event1.1342130410188.tmp for DFSClient_-1690064085 on client 0.0.0.1, because this file is already being created by DFSClient_1581651201 on 0.0.0.2
2012-07-12 22:18:51,820 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 8020, call create(/logs/event1/07122012/22/2/Event1.1342130410188.tmp, rwxr-xr-x, DFSClient_-1690064085, true, 3, 134217728) from 0.0.0.1:54280: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /logs/event1/07122012/22/2/Event1.1342130410188.tmp for DFSClient_-1690064085 on client 0.0.0.1, because this file is already being created by DFSClient_1581651201 on 0.0.0.2
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /logs/event1/07122012/22/2/Event1.1342130410188.tmp for DFSClient_-1690064085 on client 0.0.0.1, because this file is already being created by DFSClient_1581651201 on 0.0.0.2
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1338)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1178)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1126)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:585)
	at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)

Issue Links

• relates to: FLUME-1573 Duplicated HDFS file name when multiple SinkRunner was existing