[HDFS-2982] Startup performance suffers when there are many edit log segments - ASF JIRA

Voters

Stop watching

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 2.0.0-alpha
Fix Version/s: 2.0.2-alpha
Component/s: namenode
Labels:
None

Target Version/s:
Hadoop Flags:

Reviewed

Description

For every one of the edit log segments, it seems like we are calling listFiles on the edit log directory inside of findMaxTransaction. This is killing performance, especially when there are many log segments and the directory is stored on NFS. It is taking several minutes to start up the NN when there are several thousand log segments present.

Attachments

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

HDFS-2982.009.patch
21/May/12 23:58
87 kB
Colin McCabe
HDFS-2982.008.patch
21/May/12 21:35
87 kB
Colin McCabe
HDFS-2982.007.patch
21/May/12 18:38
87 kB
Colin McCabe
HDFS-2982.006.patch
21/May/12 18:13
87 kB
Colin McCabe
HDFS-2982.005.patch
18/May/12 21:08
84 kB
Colin McCabe
HDFS-2982.004.patch
18/May/12 19:02
84 kB
Colin McCabe
HDFS-2982.003.patch
18/May/12 18:18
84 kB
Colin McCabe
HDFS-2982.002.patch
18/May/12 07:14
84 kB
Colin McCabe
HDFS-2982.001.patch
18/May/12 03:19
87 kB
Colin McCabe

Issue Links

Add Link

is superceded by

HDFS-3049 During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt

Closed

Delete this link

Activity

Comment

This comment will be Viewable by All Users Viewable by All Users

Cancel

People

Assignee:: Colin McCabe

Reporter:: Todd Lipcon

Votes:: 0 Vote for this issue

Watchers:: 10 Stop watching this issue

Dates

Created:: 22/Feb/12 00:44

Updated:: 11/Oct/12 17:46

Resolved:: 23/May/12 20:43

Agile

Slack

Issue deployment