[#HADOOP-4997] workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)

Hadoop Common

workaround for tmp file handling on DataNodes in 0.18 (HADOOP-4663)

Created: 08/Jan/09 08:21 PM Updated: 08/Jul/09 04:43 PM

Return to search

Component/s:

None

Affects Version/s:

0.18.0

Fix Version/s:

0.18.3

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
deleteTmp-0.18.patch	2009-01-08 11:01 PM	dhruba borthakur	0.9 kB
HADOOP-4997-branch-18.patch	2009-01-09 12:26 AM	Raghu Angadi	4 kB

Issue Links:

Incorporates

This issue is part of:

HDFS-142 Datanode should delete files under tmp when upgraded from 0.17

Reference

This issue is related to:

~~HADOOP-5225~~ workaround for tmp file handling on DataNodes in 0.19.1 (HADOOP-4663)

Hadoop Flags:	Incompatible change
Release Note:	Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees.
Resolution Date:	16/Jan/09 11:56 PM

Description

« Hide

This is a temporary work around issues discussed in HADOOP-4663.

The proposal is to remove all the files under tmp directory, thus bringing the behavior back to 0.17. The main cost is that sync() will not be supported. This is incompatible with 0.18.x, but not with 0.17 because of this reason.

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

Raghu Angadi made changes - 08/Jan/09 08:22 PM

Field	Original Value	New Value
Link		This issue is part of HADOOP-4663 [ HADOOP-4663 ]

[ Permlink | « Hide ]

dhruba borthakur added a comment - 08/Jan/09 11:01 PM

when the datanode restarts, remove all blocks from the tmp directory of the datanode. However, fsync does not throw an exception.

dhruba borthakur made changes - 08/Jan/09 11:01 PM

Attachment

deleteTmp-0.18.patch [ 12397458 ]

[ Permlink | « Hide ]

Hairong Kuang added a comment - 08/Jan/09 11:16 PM

Dhruba, we should also remove the unit tests that test the tmp directory does not get removed after restarting the cluster.

[ Permlink | « Hide ]

Hairong Kuang added a comment - 08/Jan/09 11:47 PM

The two junit tests that would fail with this patch are TestFileCreationDelete and TestRenameWhileOpen.

[ Permlink | « Hide ]

Raghu Angadi added a comment - 09/Jan/09 12:26 AM

Thanks Hairong.

The updated patch disables the test Hairong mentioned with a very explicit comment that it is a temporary change and it has the original patch Dhruba attached.

Raghu Angadi made changes - 09/Jan/09 12:26 AM

Attachment

HADOOP-4997-branch-18.patch [ 12397470 ]

[ Permlink | « Hide ]

dhruba borthakur added a comment - 09/Jan/09 01:06 AM

Hi Hairong, I saw ur comment just now about the unit tests. Thanks for fixing it.

Raghu Angadi made changes - 14/Jan/09 08:38 PM

Assignee

Raghu Angadi [ rangadi ]

[ Permlink | « Hide ]

Konstantin Shvachko added a comment - 16/Jan/09 07:33 PM

+1 on Raghu's patch. It is critical for 0.18.

Raghu Angadi made changes - 16/Jan/09 10:01 PM

Assignee

Raghu Angadi [ rangadi ]

[ Permlink | « Hide ]

Raghu Angadi added a comment - 16/Jan/09 10:56 PM

test-patch :
------------

[exec] -1 overall.

[exec] +1 @author. The patch does not contain any @author tags.

[exec] +1 tests included. The patch appears to include 6 new or modified tests.

[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.

[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.

----------

JavaDoc failures are unrelated to this patch (they are for "CyclicIterator.java").

Repository	Revision	Date	User	Message
ASF	#735183	Fri Jan 16 23:55:37 UTC 2009	rangadi	~~HADOOP-4997~~. Revert tmp files handling on DataNodes back to 0.17.\nsync() introduced in 0.18 has less gaurantees (Raghu Angadi).
				Files Changed
				MODIFY /hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/dfs/TestRenameWhileOpen.java MODIFY /hadoop/core/branches/branch-0.18/src/hdfs/org/apache/hadoop/dfs/FSDataset.java MODIFY /hadoop/core/branches/branch-0.18/CHANGES.txt MODIFY /hadoop/core/branches/branch-0.18/src/test/org/apache/hadoop/dfs/TestFileCreationDelete.java

[ Permlink | « Hide ]

Raghu Angadi added a comment - 16/Jan/09 11:56 PM

I just committed this to 0.18 only.

Raghu Angadi made changes - 16/Jan/09 11:56 PM

Resolution		Fixed [ 1 ]
Release Note		Revert tmp files handling on DataNodes back to 0.17. sync() introduced in 0.18 has less gaurantees.
Hadoop Flags		[Incompatible change]
Status	Open [ 1 ]	Resolved [ 5 ]

[ Permlink | « Hide ]

Konstantin Shvachko added a comment - 17/Jan/09 01:47 AM

Could you please file a bug against CyclicIterator.java

[ Permlink | « Hide ]

Raghu Angadi added a comment - 17/Jan/09 02:04 AM

> Could you please file a bug against CyclicIterator.java
done : ~~HADOOP-5077~~.

Nigel Daley made changes - 30/Jan/09 08:14 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

Raghu Angadi made changes - 13/Feb/09 12:24 AM

Link

This issue incorporates ~~HADOOP-5225~~ [ ~~HADOOP-5225~~ ]

Raghu Angadi made changes - 13/Feb/09 12:31 AM

Link

This issue incorporates ~~HADOOP-5225~~ [ ~~HADOOP-5225~~ ]

Raghu Angadi made changes - 13/Feb/09 12:32 AM

Link

This issue is related to ~~HADOOP-5225~~ [ ~~HADOOP-5225~~ ]

Owen O'Malley made changes - 08/Jul/09 04:43 PM

Component/s

dfs [ 12310710 ]