Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15477

Enforce ordered snapshot deletion

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • snapshots
    • None

    Description

      Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose a configurable feature for enforcing ordered snapshot deletion – only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted.

      The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot.

      One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted.

      Attachments

        Issue Links

          1.
          Ordered snapshot deletion: make it a configurable feature Sub-task Resolved Tsz-wo Sze  
          2.
          Ordered snapshot deletion: record snapshot deletion in XAttr Sub-task Resolved Shashikant Banerjee  
          3.
          Ordered snapshot deletion: garbage collect deleted snapshots Sub-task Resolved Tsz-wo Sze  
          4.
          Add a command to list all snapshots for a snaphottable root with snapshot Ids Sub-task Resolved Shashikant Banerjee  
          5.
          Show snapshots deletion status in snapList cmd Sub-task Resolved Shashikant Banerjee  
          6.
          Ordered snapshot deletion: Disallow rename between two snapshottable directories Sub-task Resolved Shashikant Banerjee  
          7.
          Address checkstyle issues reported with HDFS-15480 Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          8.
          Make snapshot limit on global as well per snapshot root directory configurable Sub-task Resolved Shashikant Banerjee  
          9.
          Make trash root inside each snapshottable directory Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          10.
          Create trash dir when allowing snapshottable dir Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          11.
          Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          12.
          When disallowing snapshot on a dir, throw exception if its trash root is not empty Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          13.
          Disallow making a Snapshottable directory unsnapshottable if it has no empty snapshot trash directory Sub-task Resolved Siyao Meng  
          14.
          allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          15.
          Make trash root inside each snapshottable directory for WebHDFS Sub-task Resolved Siyao Meng  
          16.
          Initialize snapshot trash root during NameNode startup if enabled Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          17.
          Add UI for deleted snapshots Sub-task Resolved Vivek Ratnavel Subramanian  
          18.
          In-order deletion of snapshots: Diff lists must be update only in the last snapshot Sub-task Resolved Tsz-wo Sze

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          19.
          Wrong operation name in FsNamesystem for listSnapshots Sub-task Resolved Aryan Gupta  
          20.
          Add edit log entry for Snapshot deletion GC thread snapshot deletion Sub-task Resolved Shashikant Banerjee  
          21.
          Namenode UI: List snapshots in separate table for each snapshottable directory Sub-task Resolved Vivek Ratnavel Subramanian

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          22.
          Add identified snapshot corruption tests for ordered snapshot deletion Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          23.
          namenode start failed to start when dfs.namenode.snapshot.max.limit set Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 20m
          24.
          namenode fails to start when ordered snapshot deletion feature is disabled Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          25.
          Add list Snapshot command in WebHDFS Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          26.
          allowSnapshot fails when directory already has a Trash sub directory Sub-task Resolved Shashikant Banerjee  
          27.
          Rename snapshots while marking them deleted Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 20m
          28.
          Ensure snapshot root trash provisioning happens only post safe mode exit Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          29.
          standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 50m
          30.
          Implement dfsadmin -provisionSnapshotTrash -all Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          31.
          Metric for ordered snapshot deletion GC thread Sub-task Open Nilotpal Nandi

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          32.
          WebHDFS: Support provisionSnapshotTrash Sub-task Open Unassigned  
          33.
          Update Apache documentation for new ordered snapshot deletion feature Sub-task Open Unassigned  

          Activity

            People

              szetszwo Tsz-wo Sze
              szetszwo Tsz-wo Sze
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 29h 50m
                  29h 50m