Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14513

FSImage which is saving should be clean while NameNode shutdown

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Checkpointer/FSImageSaver is regular tasks and dump NameNode meta to disk, at most per hour by default. If it receive some command (e.g. transition to active in HA mode) it will cancel checkpoint and delete tmp files using FSImage#deleteCancelledCheckpoint. However if NameNode shutdown when checkpoint, the tmp files will not be cleaned anymore.
      Consider there are 500m inodes+blocks, it could cost 5~10min to finish once checkpoint, if we shutdown NameNode during checkpointing, fsimage checkpoint file will never be cleaned, after long time, there could be many useless checkpoint files. So I propose that we should add hook to clean that when shutdown.

        Attachments

        1. HDFS-14513.001.patch
          1 kB
          Xiaoqiao He
        2. HDFS-14513.002.patch
          2 kB
          Xiaoqiao He
        3. HDFS-14513.003.patch
          2 kB
          Xiaoqiao He
        4. HDFS-14513.004.patch
          6 kB
          Xiaoqiao He
        5. HDFS-14513.005.patch
          3 kB
          Xiaoqiao He
        6. HDFS-14513.006.patch
          3 kB
          Xiaoqiao He
        7. HDFS-14513.007.patch
          2 kB
          Xiaoqiao He

        Issue Links

          Activity

            People

            • Assignee:
              hexiaoqiao Xiaoqiao He
              Reporter:
              hexiaoqiao Xiaoqiao He

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment