[HDFS-1060] Append/flush should support concurrent "tailer" use case

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.20-append, 0.21.0, 0.22.0
• Component/s: datanode, hdfs-client
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 22/Mar/10 02:36
• Updated: 30/Jul/14 16:58
• Resolved: 30/Jul/14 16:58

Description

Several people have a usecase for a writer logging edits and using hflush() while one or more readers "tails" the file by periodically reopening and seeking to get the latest bytes. Currently there are several bugs. Using this ticket as a supertask for those bugs (and to add test cases for this use case)

Issue Links

• is related to: HDFS-265 Revisit append

Sub-Tasks

• Concurrent readers hit ChecksumExceptions if following a writer to very end of file: Closed
• reading from file under construction fails if it reader beats writer to DN for new block: Open
• completeFile loops forever if the block's only replica has become corrupt: Resolved
• client block verification may result in blocks in DataBlockScanner prematurely: Open