[HDFS-16147] load fsimage with parallelization and compression

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: namanode

Release Note

In HDFS-14617, it allows the inode and inode directory sections of the fsimage to be loaded in parallel. But it can't turn on parallelism and compression at the same time. I fixed this defect.
The idea of this patch is to compress each section separately.
1、At this patch, when compression and parallel are turned on at the same time, oiv can still work normally. Although each section is split into multiple sections and compressed, it can still be parsed correctly.
2、If you create a parallel compressed image with this patch, and then try to load it in a NN without this patch and parallel loading disabled, NN is still able to load it.
      Both of these may sound incorrect, but it works. This may happen because the decompressed code can splits each section correctly.

• Assignee: Unassigned
• Reporter: kinit
• Created: 29/Jul/21 11:46
• Updated: 05/Aug/21 00:17

Issue Links

• fixes: HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index