Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13102

Implement SnapshotSkipList class to store Multi level DirectoryDiffs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: snapshots
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-11225 explains an issue where deletion of older snapshots can take a very long time in case the no of snapshot diffs is quite large for directories. For any directory under a snapshot, to construct the children list , it needs to combine all the diffs from that particular snapshot to the last snapshotDiff record and reverseApply to the current children list of the directory on live fs. This can take  a significant time if the no of snapshot diffs are quite large and changes per diff is significant.

      This Jira proposes to store the Directory diffs in a SnapshotSkip list, where we store multi level DirectoryDiffs. At each level, the Directory Diff will be cumulative diff of k snapshot diffs,

      where k is the level of a node in the list. 

       

        Attachments

        1. HDFS-13102.001.patch
          51 kB
          Shashikant Banerjee
        2. HDFS-13102.002.patch
          27 kB
          Shashikant Banerjee
        3. HDFS-13102.003.patch
          29 kB
          Shashikant Banerjee
        4. HDFS-13102.004.patch
          33 kB
          Shashikant Banerjee
        5. HDFS-13102.005.patch
          25 kB
          Shashikant Banerjee
        6. HDFS-13102.006.patch
          27 kB
          Shashikant Banerjee
        7. HDFS-13102.007.patch
          28 kB
          Shashikant Banerjee
        8. HDFS-13102.008.patch
          26 kB
          Shashikant Banerjee
        9. HDFS-13102.009.patch
          26 kB
          Shashikant Banerjee
        10. HDFS-13102.009_committed.patch
          26 kB
          Tsz Wo Nicholas Sze

          Issue Links

            Activity

              People

              • Assignee:
                shashikant Shashikant Banerjee
                Reporter:
                shashikant Shashikant Banerjee
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: