[HDFS-265] Revisit append

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0
• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Created: 24/Apr/09 21:06
• Updated: 07/Jan/16 18:37
• Resolved: 29/Mar/10 23:30

Description

HADOOP-1700 and related issues have put a lot of efforts to provide the first implementation of append. However, append is such a complex feature. It turns out that there are issues that were initially seemed trivial but needs a careful design. This jira revisits append, aiming for a design and implementation supporting a semantics that are acceptable to its users.

Issue Links

• incorporates: HDFS-728 Create a comprehensive functional test for append
• is related to: HADOOP-6239 Command-line for append
• is related to: HDFS-744 Support hsync in HDFS
• is related to: HDFS-1831 HDFS equivalent of HADOOP-7223 changes to handle FileContext createFlag combinations
• relates to: HDFS-588 Fix TestFiDataTransferProtocol and TestAppend2 failures in append branch.
• relates to: HDFS-1060 Append/flush should support concurrent "tailer" use case
• relates to: HDFS-195 Need to handle access token expiration when re-establishing the pipeline for dfs write
• relates to: HDFS-1606 Provide a stronger data guarantee in the write pipeline
• relates to: HADOOP-4359 Access Token: Support for data access authorization checking on DataNodes
• relates to: HDFS-714 Create fault injection test for the new pipeline close
• relates to: HDFS-451 Test DataTransferProtocol with fault injection
• supercedes: HADOOP-1700 Append to files in HDFS

Sub-Tasks

• Factor out BlockInfo from BlocksMap: Closed
• Redesign DataNode volumeMap to include all types of Replicas: Resolved
• Introduce BlockInfoUnderConstruction to reflect block replica states while writing.: Resolved
• Create new tests for Append's hflush: Resolved
• Create new tests for lease recovery: Closed
• Create new tests for block recovery: Closed
• Create new tests for pipeline: Closed
• Create stress tests for append feature: Resolved
• Support hflush at DFSClient: Resolved
• DataNode uses ReplicaBeingWritten to support dfs writes/hflush: Resolved
• Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append: Resolved
• Add a "rbw" sub directory to DataNode data directory: Resolved
• DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas: Resolved
• Add a test for NameNode.getBlockLocations(..) to check read from un-closed file.: Resolved
• Introduce block committing logic during new block allocation and file close.: Resolved
• When opening a file for read, make the file length avaliable to client.: Resolved
• Extend Block report to include under-construction replicas: Resolved
• Datanode should serve up to visible length of a replica for read requests: Resolved
• Change block write protocol to support pipeline recovery: Resolved
• Create new functional test for a block report.: Closed
• Allow client to get a new generation stamp from NameNode: Resolved
• Block report processing for append: Resolved
• Create functional tests for new design of the block report: Closed
• Support replica recovery initialization in datanode: Resolved
• Client support pipeline recovery: Resolved
• Support replica update in datanode: Resolved
• SafeMode should count only complete blocks.: Resolved
• Support pipeline close and close recovery: Resolved
• Lease recovery, concurrency support.: Resolved
• Replace BlockInfo.isUnderConstruction() with isComplete(): Resolved
• Simplify the codes in the replica related classes: Resolved
• Remove unused legacy protocol methods.: Resolved
• Block recovery for primary data-node: Resolved
• DFSClient cannot read all the available bytes: Resolved
• Remove deprecated methods from InterDatanodeProtocol.: Closed
• Data-node upgrade problem: Resolved
• Unnecessary info message from DFSClient: Resolved
• DFSIO for append: Closed
• TestFileAppend2 sometimes hangs: Resolved
• TestFileAppend3#TC7 sometimes hangs: Closed
• NPE in FSDataset.updateReplicaUnderRecovery(..): Closed
• Create block recovery tests that handle errors: Closed