[HDFS-2126] Improve Namenode startup time [umbrella task] - ASF JIRA

XML

Word

Printable

JSON

This is an umbrella task to group the improvements in Namenode startup latency made over the last few months, and track remaining ideas.

depends upon

HADOOP-6949 Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting

is blocked by

HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file

relates to

HDFS-1910 when dfs.name.dir and dfs.name.edits.dir are same fsimage will be saved twice every time

1.	Improve namenode restart times by short-circuiting the first block reports from datanodes	Resolved	Matthew Foley
2.	Enhance NNThroughputBenchmark to observe scale-dependent changes in IBR processing	Open	Matthew Foley
3.	Not marking datanodes dead When namenode in safemode	Closed	Hairong Kuang
4.	Namenode should ignore non-initial block reports from datanodes when in safemode during startup	Closed	Matthew Foley
5.	savenamespace should write the fsimage to all configured fs.name.dir in parallel	Closed	Dmytro Molkov
6.	NameNode should save image to name directories in parallel during upgrade	Closed	Matthew Foley
7.	Speedup NameNode image loading and saving by storing local file names	Resolved	Hairong Kuang
8.	Exiting safemode takes a long time when there are lots of blocks in the HDFS	Open	Dhruba Borthakur