Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
Reviewed
Description
Add support to save Inodes stored in PartitionedGSet when saving an FS image and load Inodes into PartitionedGSet from a saved FS image.
Saving FSImage
Original HDFS design: iterate every inode in inodeMap and save them into the FSImage file.
FGL: no change is needed here, since PartitionedGSet also provides an iterator interface, to iterate over inodes stored in partitions.
Loading an HDFS
Original HDFS design: it first loads the FSImage files and then loads edit logs for recent changes. FSImage files contain different sections, including INodeSections and INodeDirectorySections. An InodeSection contains serialized Inodes objects and the INodeDirectorySection contains the parent inode for an Inode. When loading an FSImage, the system first loads INodeSections and then load the INodeDirectorySections, to set the parent inode for each inode.
After FSImage files are loaded, edit logs are then loaded. Edit log contains recent changes to the filesystem, including Inodes creation/deletion. For a newly created INode, the parent inode is set before it is added to the inodeMap.
FGL: when adding an Inode into the partitionedGSet, we need the parent inode of an inode, in order to determine which partition to store that inode, when NAMESPACE_KEY_DEPTH = 2. Thus, in FGL, when loading FSImage files, we used a temporary LightweightGSet (inodeMapTemp), to store inodes. When LoadFSImage is done, the parent inode for all existing inodes in FSImage files is set. We can now move the inodes into a partitionedGSet. Load edit logs can work as usual, as the parent inode for an inode is set before it is added to the inodeMap.
In theory, PartitionedGSet can support to store inodes without setting its parent inodes. All these inodes will be stored in the 0th partition. However, we decide to use a temporary LightweightGSet (inodeMapTemp) to store these inodes, to make this case more transparent.
Attachments
Issue Links
- links to
