[HDFS-3998] Speed up fsck

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: namenode
• Assignee: Unassigned
• Reporter: Ming Ma
• Created: 02/Oct/12 21:59
• Updated: 10/Nov/12 23:34

Description

We have some big clusters. Sometimes we want to find out the list of missing blocks or blocks with only one replica quickly. Currently fsck has to take a path as input and it then recursively check for inconsistency. That could take a long time to find the missing blocks and the files the missing blocks belong to. It will be useful to speed this up. For example, it could go directly to missing blocks stored in NN and do the file lookup instead.

Issue Links

• is related to: HDFS-3990 NN's health report has severe performance problems
• relates to: HDFS-2554 Add separate metrics for missing blocks with desired replication level 1
• relates to: HDFS-1031 Enhance the webUi to list a few of the corrupted files in HDFS
• relates to: HDFS-4174 Backport HDFS-1031 to branch-1: to list a few of the corrupted files in WebUI