Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12352

Delay in checkpointing Trash can leave trash for 2 intervals before deleting

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: trash
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixes an Trash related issue wherein a delay in the periodic checkpointing of one user's directory causes the subsequent user directory checkpoints to carry a newer timestamp, thereby delaying their eventual deletion.

      Description

      When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and the same, it is possible for trash to be left for two intervals.

      The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash will be checkpointed every "interval" of minutes.

      Each user's trash is checkpointed individually. The time resolution of the checkpoint timestamp is to the second.

      If the seconds switch while one user is checkpointing, then the next user's timestamp will be later.

      This will cause the next user's checkpoint to not be deleted at the next interval.

      I have recreated this in a lab cluster

      I also have a suggestion for a patch that I can upload later tonight after testing it further.

        Attachments

        1. HDFS-8118.001.patch
          2 kB
          Casey Brotherton
        2. HDFS-8118.patch
          2 kB
          Casey Brotherton

          Activity

            People

            • Assignee:
              caseyjbrotherton Casey Brotherton
              Reporter:
              caseyjbrotherton Casey Brotherton
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: