[#HADOOP-3310] Lease recovery for append

Hadoop Common

Lease recovery for append

Created: 24/Apr/08 11:43 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.18.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
3310_20080514.patch	2008-05-14 05:25 PM	Tsz Wo (Nicholas), SZE	38 kB
3310_20080516b.patch	2008-05-16 10:38 PM	Tsz Wo (Nicholas), SZE	49 kB
3310_20080516c.patch	2008-05-16 10:55 PM	Tsz Wo (Nicholas), SZE	47 kB
3310_20080519.patch	2008-05-19 08:49 PM	Tsz Wo (Nicholas), SZE	51 kB
3310_20080519b.patch	2008-05-20 12:52 AM	Tsz Wo (Nicholas), SZE	62 kB
3310_20080520.patch	2008-05-21 01:21 AM	Tsz Wo (Nicholas), SZE	76 kB
3310_20080521.patch	2008-05-21 07:19 PM	Tsz Wo (Nicholas), SZE	78 kB
3310_20080522b.patch	2008-05-22 09:20 PM	Tsz Wo (Nicholas), SZE	81 kB
3310_20080522c.patch	2008-05-22 10:13 PM	Tsz Wo (Nicholas), SZE	82 kB
3310_20080523.patch	2008-05-24 02:15 AM	Tsz Wo (Nicholas), SZE	85 kB
3310_20080524_dhruba.patch	2008-05-24 09:22 AM	dhruba borthakur	88 kB
3310_20080527.patch	2008-05-28 01:10 AM	Tsz Wo (Nicholas), SZE	100 kB
3310_20080528.patch	2008-05-28 08:21 PM	Tsz Wo (Nicholas), SZE	102 kB
3310_20080528b.patch	2008-05-28 09:54 PM	Tsz Wo (Nicholas), SZE	101 kB
3310_20080528c.patch	2008-05-29 12:50 AM	Tsz Wo (Nicholas), SZE	104 kB
3310_20080529.patch	2008-05-29 08:00 AM	dhruba borthakur	96 kB
3310_20080529b.patch	2008-05-29 06:19 PM	dhruba borthakur	105 kB
3310_20080529c.patch	2008-05-29 10:07 PM	Tsz Wo (Nicholas), SZE	108 kB

Issue Links:

Blocker

This issue blocks:

~~HADOOP-3113~~ DFSOututStream.flush() should flush data to real block file on DataNode.

This issue is blocked by:

~~HADOOP-3375~~ Lease paths are sometimes not removed from LeaseManager.sortedLeasesByPath

Dependants

This issue blocks:

~~HADOOP-1700~~ Append to files in HDFS

Hadoop Flags:	Reviewed, Incompatible change
Release Note:	Implemented Lease Recovery to sync the last bock of a file. Added ClientDatanodeProtocol for client trigging block recovery. Changed DatanodeProtocol to support block synchronization. Changed InterDatanodeProtocol to support block update.
Resolution Date:	02/Jun/08 06:46 PM

Description

« Hide

In order to support file append, a GenerationStamp is associated with each block. Lease recovery will be performed when there is a possibility that the replicas of a block in a lease may have different GenerationStamp values.

For more details, see the documentation in ~~HADOOP-1700~~.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Repository	Revision	Date	User	Message
ASF	#662513	Mon Jun 02 18:43:01 UTC 2008	dhruba	~~HADOOP-3310~~. The namenode instructs the primary datanode to do lease recovery. The block gets a new generation stamp. (Tsz Wo (Nicholas), SZE via dhruba)
				Files Changed
				MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DataNode.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DatanodeBlockInfo.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSNamesystem.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DatanodeID.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSImage.java ADD /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestLeaseRecovery.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSEditLog.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/Block.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DatanodeProtocol.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSDataset.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/INode.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/GenerationStamp.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/BlockMetaDataInfo.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/BlockCommand.java ADD /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestFileCreationNamenodeRestart.java MODIFY /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestFileCreation.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DatanodeDescriptor.java MODIFY /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/SimulatedFSDataset.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSDirectory.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/InterDatanodeProtocol.java MODIFY /hadoop/core/trunk/CHANGES.txt MODIFY /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestInterDatanodeProtocol.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/DFSClient.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/LeaseManager.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/StringBytesWritable.java MODIFY /hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestGetBlocks.java ADD /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/ClientDatanodeProtocol.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/BlocksMap.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/FSDatasetInterface.java MODIFY /hadoop/core/trunk/src/java/org/apache/hadoop/dfs/NameNode.java

Repository	Revision	Date	User	Message
ASF	#662515	Mon Jun 02 18:45:16 UTC 2008	dhruba	~~HADOOP-3310~~. The namenode instructs the primary datanode to do lease recovery. The block gets a new generation stamp. Tsz Wo (Nicholas), SZE via dhruba)
				Files Changed
				MODIFY /hadoop/core/trunk/CHANGES.txt