Details

    • Type: Wish Wish
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.23.0
    • Fix Version/s: None
    • Component/s: hdfs-client, namenode
    • Labels:

      Description

      Currently trash is disabled out of box. I do not think it'd be of high surprise to anyone (but surely a relief when *hit happens) to have trash enabled by default, with the usually recommended periods of 1-day.

      Thoughts?

      1. hdfs-2740.patch
        4 kB
        T Meyarivan
      2. hdfs-2740.patch
        0.8 kB
        T Meyarivan

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          The decision here is to not change this default behavior.

          However, we can still improve the docs, for which the JIRA is now available at HDFS-3302

          Show
          Harsh J added a comment - The decision here is to not change this default behavior. However, we can still improve the docs, for which the JIRA is now available at HDFS-3302
          Hide
          Harsh J added a comment -

          I think 1-hour is too short. Sometimes you don't even realize you'd lost something until hours later (I'm a victim myself). If we are enabling, we should enable for a day at minimum. If the client chooses not to have trash, he can override it himself.

          @Others - Comments on proceeding? I'm inclined to commit this in with some more documented Q&A added, but since it has run this wide and has gathered a few watchers, I'd like your final opinions (+/-).

          Show
          Harsh J added a comment - I think 1-hour is too short. Sometimes you don't even realize you'd lost something until hours later (I'm a victim myself). If we are enabling, we should enable for a day at minimum. If the client chooses not to have trash, he can override it himself. @Others - Comments on proceeding? I'm inclined to commit this in with some more documented Q&A added, but since it has run this wide and has gathered a few watchers, I'd like your final opinions (+/-).
          Hide
          T Meyarivan added a comment -

          Re the common case where a user issues an incorrect delete cmd - Is short fs.trash.interval (even as low as 1 hr) reasonable ? (recovery is still possible, the space is freed up in a reasonable time frame for others)

          Show
          T Meyarivan added a comment - Re the common case where a user issues an incorrect delete cmd - Is short fs.trash.interval (even as low as 1 hr) reasonable ? (recovery is still possible, the space is freed up in a reasonable time frame for others) –
          Hide
          Harsh J added a comment -

          Eli Collins - The FsShell does log out that the file was moved to trash and not completely removed. If we can solve this with more info/doc efforts, am up for doing that.

          I do think a lot of them miss out on the trash feature until they run into a situation that makes them search if there is one.

          Stuff we can document more explicitly about, to help:

          • How do I disable Trash?
          • How do I clear out Trash?
          • How do I force-delete a file (skipping trash)?
          • How do I tweak the checkpoint periods?

          And maybe some dev documentation on trash policies, as I think that is now pluggable (evolving API)?

          Show
          Harsh J added a comment - Eli Collins - The FsShell does log out that the file was moved to trash and not completely removed. If we can solve this with more info/doc efforts, am up for doing that. I do think a lot of them miss out on the trash feature until they run into a situation that makes them search if there is one. Stuff we can document more explicitly about, to help: How do I disable Trash? How do I clear out Trash? How do I force-delete a file (skipping trash)? How do I tweak the checkpoint periods? And maybe some dev documentation on trash policies, as I think that is now pluggable (evolving API)?
          Hide
          Eli Collins added a comment -

          I'm not sold that we should ship with Trash enabled out of the box. Equally confusing is users who expect deleting files actually frees up space right?

          Show
          Eli Collins added a comment - I'm not sold that we should ship with Trash enabled out of the box. Equally confusing is users who expect deleting files actually frees up space right?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12510051/hdfs-2740.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1770//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510051/hdfs-2740.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1770//console This message is automatically generated.
          Hide
          T Meyarivan added a comment -

          Includes changes to docs.

          Show
          T Meyarivan added a comment - Includes changes to docs.
          Hide
          Harsh J added a comment -

          Meyarivan,

          Your changeset is fine. HDFS buildbot failed to apply it cause its under the hadoop-common directory, but I think the JIRA should stick here, on HDFS, as its relevant to this component.

          Could you also update the docs regarding this behavior change, and incorporate Daryn Sharp's and Robert Joseph Evans's comments above into the docs?

          The trash feature is currently documented at hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_design.xml, but if you feel its fit in the filesystem_shell guide itself, feel free to move it there.

          Show
          Harsh J added a comment - Meyarivan, Your changeset is fine. HDFS buildbot failed to apply it cause its under the hadoop-common directory, but I think the JIRA should stick here, on HDFS, as its relevant to this component. Could you also update the docs regarding this behavior change, and incorporate Daryn Sharp 's and Robert Joseph Evans 's comments above into the docs? The trash feature is currently documented at hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_design.xml , but if you feel its fit in the filesystem_shell guide itself, feel free to move it there.
          Hide
          T Meyarivan added a comment -

          Re config changes, do they(for params fs.trash.interval, fs.trash.checkpoint.interval) need to be present in :

          [a] both the files ? (will create a new bug and apply diff to core-default.xml, apply diff to hdfs-default.xml)
          [b] only core-default.xml ? (will create a new bug, apply diff to core-default.xml, close this bug as 'duplicate')
          [c] only hdfs-default.xml ? (will remove entries from core-default.xml via new bug, apply diff to hdfs-default.xml)

          Show
          T Meyarivan added a comment - Re config changes, do they(for params fs.trash.interval, fs.trash.checkpoint.interval) need to be present in : [a] both the files ? (will create a new bug and apply diff to core-default.xml, apply diff to hdfs-default.xml) [b] only core-default.xml ? (will create a new bug, apply diff to core-default.xml, close this bug as 'duplicate') [c] only hdfs-default.xml ? (will remove entries from core-default.xml via new bug, apply diff to hdfs-default.xml) –
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12509711/hdfs-2740.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1764//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509711/hdfs-2740.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1764//console This message is automatically generated.
          Hide
          T Meyarivan added a comment -

          Patch introduces following changes:

          [1] Create new snapshot every ~360 minutes (6 hours)
          [2] Retain deleted data for ~1440 minutes (24 hours)

          Show
          T Meyarivan added a comment - Patch introduces following changes: [1] Create new snapshot every ~360 minutes (6 hours) [2] Retain deleted data for ~1440 minutes (24 hours)
          Hide
          Robert Joseph Evans added a comment -

          I think that quotas are also off by default, but yes anyone upgrading that has quotas on and has not overridden the trash setting in their -site.xml would potentially see this. I think most of that can be overcome with documentation though.

          Show
          Robert Joseph Evans added a comment - I think that quotas are also off by default, but yes anyone upgrading that has quotas on and has not overridden the trash setting in their -site.xml would potentially see this. I think most of that can be overcome with documentation though.
          Hide
          Daryn Sharp added a comment -

          It's worth considering that changing the default may cause current deployments to experience unexpected quota exceeded exceptions.

          Show
          Daryn Sharp added a comment - It's worth considering that changing the default may cause current deployments to experience unexpected quota exceeded exceptions.
          Hide
          Harsh J added a comment -

          Coupled with a documentation effort that makes it more visible should help further.

          Show
          Harsh J added a comment - Coupled with a documentation effort that makes it more visible should help further.

            People

            • Assignee:
              Unassigned
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development