[HDFS-1435] Provide an option to store fsimage compressed

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.22.0
• Fix Version/s: 0.22.0
• Component/s: namenode
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.
• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Created: 30/Sep/10 23:56
• Updated: 09/Jul/12 21:28
• Resolved: 24/Oct/10 04:52

Description

Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.

If we could store fsimage compressed, the problem could be greatly alleviated.

I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.

The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.

Issue Links

• incorporates: HDFS-1614 Provide an option to saveNamespace to save namespace uncompressed
• is blocked by: HADOOP-6996 Allow CodecFactory to return a codec object given a codec' class name
• is related to: HDFS-1458 Improve checkpoint performance by avoiding unnecessary image downloads
• is related to: HDFS-1457 Limit transmission rate when transfering image between primary and secondary NNs
• relates to: HDFS-1465 Eliminate FS image loading code duplication between OIV and FSImage