[HADOOP-6467] Performance improvement for liststatus on directories in hadoop archives. - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.21.0
Component/s: fs
Labels:
None

Hadoop Flags:

Reviewed

Description

A liststatus call on a directory in hadoop archives leads to ( 2* number of files in directory) open calls to the namenode. This is very sub optimal and needs to be fixed to make it performant enough to be used on a daily basis.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

Archives_performance.docx
29/Jan/10 00:28
111 kB
Mahadev Konar
Archives_performance.docx
23/Dec/09 22:51
94 kB
Mahadev Konar
HADOOP-6467_v3.patch
23/Feb/10 01:02
4 kB
Mahadev Konar
HADOOP-6467.patch
20/Feb/10 01:58
4 kB
Mahadev Konar
HADOOP-6467.patch
18/Feb/10 19:56
7 kB
Mahadev Konar
HADOOP-6467.patch
13/Feb/10 06:00
6 kB
Mahadev Konar
HADOOP-6467-v2.patch
22/Feb/10 22:25
4 kB
Mahadev Konar
HADOOP-6467-y.0.20-branch-v2.patch
23/Feb/10 00:47
5 kB
Mahadev Konar
HADOOP-6467-y.0.20-branch-v2.patch
22/Feb/10 22:15
4 kB
Mahadev Konar
HADOOP-6467-y0.20-branch.patch
20/Feb/10 01:57
4 kB
Mahadev Konar

Issue Links

is related to

MAPREDUCE-1629 Get rid of fakeBlockLocations() on HarFileSystem, since it's not used

Closed

relates to

HADOOP-6591 HarFileSystem cannot handle paths with the space character

Closed

Activity

People

Assignee:: Mahadev Konar

Reporter:: Mahadev Konar

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Dates

Created:: 23/Dec/09 18:31

Updated:: 24/Aug/10 20:41

Resolved:: 23/Feb/10 03:59