[HADOOP-1765] IOException on close: Unknown file

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Fix Version/s: 0.15.0
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 22/Aug/07 23:41
• Updated: 08/Jul/09 16:41
• Resolved: 24/Aug/07 06:21

Description

On occasion I get an IOE trying to close a MapFile I've just opened and filled. Below is the content of the namenode log. The first thing in it after startup messages is the IOE followed by other messages about '...current leaseholder is trying to recreate file'. It repeats for ever. This is TRUNK r567876.

I filed this issue at Raghu's suggestion. Here's what he said on the list:

.bq Yes, the earler message was also from Namenode log. Before that line there should have been a message that adds a block to this file in the same log. I think you should file a Jira for this. It probably related to HADOOP-999. Raghu.

2007-08-22 21:38:49,071 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/208.76.44.139:50010
2007-08-22 21:43:50,216 INFO org.apache.hadoop.fs.FSNamesystem: Roll Edit Log from 208.76.44.139
2007-08-22 21:45:21,459 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call complete(/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data, DFSClient_1857293290) from 208.76.44.139:52301: error: java.io.IOException: Unknown file: /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
java.io.IOException: Unknown file: /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
    at org.apache.hadoop.dfs.FSDirectory.addBlocks(FSDirectory.java:561)
    at org.apache.hadoop.dfs.FSNamesystem.completeFileInternal(FSNamesystem.java:1002)
    at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:952)
    at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:348)
    at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
2007-08-22 21:45:24,394 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.startFile: failed to create file /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder is trying to recreate file.
2007-08-22 21:45:24,394 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000, call create(/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data, DFSClient_1857293290, true, 3, 67108864) from 208.76.44.139:52312: error: org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create file /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder is trying to recreate file.
org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create file /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder is trying to recreate file.
    at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:740)
    at org.apache.hadoop.dfs.NameNode.create(NameNode.java:307)
    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

Let me try and replicate with DEBUG level enabled. Also, I think I know how to easily replicate. Need to do some tests.

Issue Links

• is blocked by: HADOOP-1708 make files visible in the namespace as soon as they are created