Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15477

Enforce ordered snapshot deletion

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: snapshots
    • Labels:
      None

      Description

      Snapshot deletion has caused a few bugs earlier such as HDFS-13101 and HDFS-15313. In this JIRA, we propose a configurable feature for enforcing ordered snapshot deletion – only the earliest snapshot is actually deleted from the file system. The other snapshots are only marked as deleted. They will be actually deleted from the file system until all the earlier snapshots are deleted.

      The reason of enforcing ordered snapshot deletion is based on the observation that the logic of deleting the earliest snapshot is much simpler since the prior snapshot does not exist. All the previous bugs are caused by removing inodes from the prior snapshot.

      One drawback of ordered snapshot deletion is that the resources are not yet released for deleting the non-earliest snapshots since they are only marked as deleted but not actually deleted.

        Attachments

          Issue Links

          1.
          Ordered snapshot deletion: make it a configurable feature Sub-task Resolved Tsz-wo Sze  
          2.
          Ordered snapshot deletion: record snapshot deletion in XAttr Sub-task Resolved Shashikant Banerjee  
          3.
          Ordered snapshot deletion: garbage collect deleted snapshots Sub-task Resolved Tsz-wo Sze  
          4.
          Add a command to list all snapshots for a snaphottable root with snapshot Ids Sub-task Resolved Shashikant Banerjee  
          5.
          Show snapshots deletion status in snapList cmd Sub-task Resolved Shashikant Banerjee  
          6.
          Ordered snapshot deletion: Disallow rename between two snapshottable directories Sub-task Resolved Shashikant Banerjee  
          7.
          Address checkstyle issues reported with HDFS-15480 Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          8.
          Make snapshot limit on global as well per snapshot root directory configurable Sub-task Resolved Shashikant Banerjee  
          9.
          Make trash root inside each snapshottable directory Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          10.
          Create trash dir when allowing snapshottable dir Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          11.
          Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          12.
          When disallowing snapshot on a dir, throw exception if its trash root is not empty Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          13.
          Disallow making a Snapshottable directory unsnapshottable if it has no empty snapshot trash directory Sub-task Resolved Siyao Meng  
          14.
          allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2.5h
          15.
          Make trash root inside each snapshottable directory for WebHDFS Sub-task Resolved Siyao Meng  
          16.
          Initialize snapshot trash root during NameNode startup if enabled Sub-task Resolved Siyao Meng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 10m
          17.
          WebHDFS: Support provisionSnapshotTrash Sub-task Open Unassigned  
          18.
          Add UI for deleted snapshots Sub-task Resolved Vivek Ratnavel Subramanian  
          19.
          In-order deletion of snapshots: Diff lists must be update only in the last snapshot Sub-task Resolved Tsz-wo Sze

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          20.
          Update Apache documentation for new ordered snapshot deletion feature Sub-task Open Unassigned  
          21.
          Wrong operation name in FsNamesystem for listSnapshots Sub-task Resolved Aryan Gupta  
          22.
          Add edit log entry for Snapshot deletion GC thread snapshot deletion Sub-task Resolved Shashikant Banerjee  
          23.
          Namenode UI: List snapshots in separate table for each snapshottable directory Sub-task Resolved Vivek Ratnavel Subramanian

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          24.
          Add identified snapshot corruption tests for ordered snapshot deletion Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          25.
          namenode start failed to start when dfs.namenode.snapshot.max.limit set Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 3h 20m
          26.
          namenode fails to start when ordered snapshot deletion feature is disabled Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          27.
          Add list Snapshot command in WebHDFS Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          28.
          Metric for ordered snapshot deletion GC thread Sub-task Open Nilotpal Nandi

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          29.
          allowSnapshot fails when directory already has a Trash sub directory Sub-task Resolved Shashikant Banerjee  
          30.
          Rename snapshots while marking them deleted Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 20m
          31.
          Ensure snapshot root trash provisioning happens only post safe mode exit Sub-task Resolved Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1.5h
          32.
          standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories Sub-task Open Shashikant Banerjee

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 10m

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                szetszwo Tsz-wo Sze
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 26h 10m
                  26h 10m