Resolution: Won't Fix
Affects Version/s: 0.95.2
Fix Version/s: None
This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira for this.
Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes, the file that was written is read by other processes to replay the action.
- So we have in hdfs one (dead) process writing with another process reading.
- But, despite the call to syncFs, we don't always see the data when we have a dead node. It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set the length to 0.
- So we may miss all the writes to the last block if we try to connect to the dead DN.
hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
hdfs branch-2 or trunk: we should not have the issue (but not tested)
The attached test will fail ~50 of the time.