[#HADOOP-2565] DFSPath cache of FileStatus can become stale

Hadoop Common

DFSPath cache of FileStatus can become stale

Created: 09/Jan/08 06:53 PM Updated: 08/Jul/09 04:42 PM

Return to search

Component/s:

None

Affects Version/s:

0.16.0

Fix Version/s:

0.18.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
2565_20080424.patch	2008-04-25 01:49 AM	Tsz Wo (Nicholas), SZE	5 kB
2565_20080604.patch	2008-06-04 09:13 PM	Tsz Wo (Nicholas), SZE	4 kB
2565_20080604b.patch	2008-06-04 09:52 PM	Tsz Wo (Nicholas), SZE	4 kB
2565_20080605.patch	2008-06-05 08:33 PM	Tsz Wo (Nicholas), SZE	4 kB

Issue Links:

Blocker

This issue blocks:

~~HADOOP-1700~~ Append to files in HDFS

This issue is blocked by:

~~HADOOP-2566~~ need FileSystem#globStatus method

Dependants

This issue depends on:

~~HADOOP-2563~~ Remove deprecated FileSystem#listPaths()

~~HADOOP-3095~~ Validating input paths and creating splits is slow on S3

Hadoop Flags:	Reviewed
Resolution Date:	05/Jun/08 08:58 PM

Description

« Hide

Paths returned from DFS internally cache their FileStatus, so that getStatus(Path) does not require another RPC. This cache is never refreshed and become stale, resulting in program error.

This should not be fixed until FileSystem#listStatus() is removed by ~~HADOOP-2563~~, and user code is thus no longer dependent on this cache for good performance.