Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14513

FSImage which is saving should be clean while NameNode shutdown

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Checkpointer/FSImageSaver is regular tasks and dump NameNode meta to disk, at most per hour by default. If it receive some command (e.g. transition to active in HA mode) it will cancel checkpoint and delete tmp files using FSImage#deleteCancelledCheckpoint. However if NameNode shutdown when checkpoint, the tmp files will not be cleaned anymore.
      Consider there are 500m inodes+blocks, it could cost 5~10min to finish once checkpoint, if we shutdown NameNode during checkpointing, fsimage checkpoint file will never be cleaned, after long time, there could be many useless checkpoint files. So I propose that we should add hook to clean that when shutdown.

        Attachments

        1. HDFS-14513.001.patch
          1 kB
          Xiaoqiao He
        2. HDFS-14513.002.patch
          2 kB
          Xiaoqiao He
        3. HDFS-14513.003.patch
          2 kB
          Xiaoqiao He
        4. HDFS-14513.004.patch
          6 kB
          Xiaoqiao He
        5. HDFS-14513.005.patch
          3 kB
          Xiaoqiao He
        6. HDFS-14513.006.patch
          3 kB
          Xiaoqiao He
        7. HDFS-14513.007.patch
          2 kB
          Xiaoqiao He

          Issue Links

            Activity

              People

              • Assignee:
                hexiaoqiao Xiaoqiao He
                Reporter:
                hexiaoqiao Xiaoqiao He
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: