Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Provide a new option to rm and rmr, -skipTrash, which will immediately delete the files specified, rather than moving them to the trash.

      Description

      Currently with quota turned on, user cannot call '-rmr' on large directory that causes over quota.

      [knoguchi src]$ hadoop dfs -rmr /tmp/net2
      rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
      [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
      mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/knoguchi is exceeded: namespace
      quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
      

      Besides from error message being unfriendly, how should this be handled?

      1. javac_warnings_diff.txt
        17 kB
        Jakob Homan
      2. HADOOP-6080-v20.patch
        9 kB
        Jakob Homan
      3. HADOOP-6080.patch
        9 kB
        Jakob Homan

        Issue Links

          Activity

          Koji Noguchi created issue -
          Hide
          Koji Noguchi added a comment -

          1) Tell users to use -Dfs.trash.interval=0 when deleting large directory
          2) Exclude /user/<username>/.Trash from the quota
          3) Move .Trash out of /user directory. Maybe /Trash/<username> and set different quota.
          4) When -rm/rmr fail with quota, automatically delete them.
          5) Introduce a separate command that does (1). Something like -rmr -skipTrash for force delete.

          Show
          Koji Noguchi added a comment - 1) Tell users to use -Dfs.trash.interval=0 when deleting large directory 2) Exclude /user/<username>/.Trash from the quota 3) Move .Trash out of /user directory. Maybe /Trash/<username> and set different quota. 4) When -rm/rmr fail with quota, automatically delete them. 5) Introduce a separate command that does (1). Something like -rmr -skipTrash for force delete.
          Jakob Homan made changes -
          Field Original Value New Value
          Assignee Jakob Homan [ jghoman ]
          Hide
          Raghu Angadi added a comment -

          +1 for (5).

          Show
          Raghu Angadi added a comment - +1 for (5).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          If we do (5), we still have to do (1), i.e. add a message telling the user to use -skipTrash .

          Show
          Tsz Wo Nicholas Sze added a comment - If we do (5), we still have to do (1), i.e. add a message telling the user to use -skipTrash .
          Hide
          Jakob Homan added a comment -

          I'm going to go ahead and implement 5.

          Show
          Jakob Homan added a comment - I'm going to go ahead and implement 5.
          Hide
          Jakob Homan added a comment -

          Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which performs as expected. Adds to trash unit test to verify correct execution. Changes documentation to reflect new option. Docs suggest this option as being a solution when a directory is over quota.

          Passes all commons unit tests. Running test patch now. Will post those results when done.

          Show
          Jakob Homan added a comment - Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which performs as expected. Adds to trash unit test to verify correct execution. Changes documentation to reflect new option. Docs suggest this option as being a solution when a directory is over quota. Passes all commons unit tests. Running test patch now. Will post those results when done.
          Jakob Homan made changes -
          Attachment HADOOP-6080.patch [ 12412112 ]
          Hide
          Jakob Homan added a comment -

          submitting patch.

          Show
          Jakob Homan added a comment - submitting patch.
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Hide
          Jakob Homan added a comment -

          Test-patch:

          [exec] -1 overall.  
          [exec] 
          [exec]     +1 @author.  The patch does not contain any @author tags.
          [exec] 
          [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
          [exec] 
          [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
          [exec] 
          [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
          [exec] 
          [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
          [exec] 
          [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          

          This is weird. I've attached the javac warnings it says are new and they have nothing to do with this patch. test-patch must be broken in this regard. I believe the patch is ready to go.

          Show
          Jakob Homan added a comment - Test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. This is weird. I've attached the javac warnings it says are new and they have nothing to do with this patch. test-patch must be broken in this regard. I believe the patch is ready to go.
          Jakob Homan made changes -
          Attachment javac_warnings_diff.txt [ 12412115 ]
          Hide
          Jakob Homan added a comment -

          Attaching two new files:

          • Updated patch. Previous patch missed updating the help text for rmr to include -skipTrash option. No change to actual code.
          • Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to be changed for patch, just file locations were different. Code is still the same. Passes unit tests.
          Show
          Jakob Homan added a comment - Attaching two new files: Updated patch. Previous patch missed updating the help text for rmr to include -skipTrash option. No change to actual code. Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to be changed for patch, just file locations were different. Code is still the same. Passes unit tests.
          Jakob Homan made changes -
          Attachment HADOOP-6080.patch [ 12412123 ]
          Attachment HADOOP-6080-v20.patch [ 12412124 ]
          Hide
          Jakob Homan added a comment -

          Canceling patch to double check something.

          Show
          Jakob Homan added a comment - Canceling patch to double check something.
          Jakob Homan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jakob Homan made changes -
          Attachment HADOOP-6080-v20.patch [ 12412124 ]
          Jakob Homan made changes -
          Attachment HADOOP-6080.patch [ 12412123 ]
          Jakob Homan made changes -
          Attachment HADOOP-6080.patch [ 12412112 ]
          Hide
          Jakob Homan added a comment -

          Ran into a problem that I didn't notice with the HDFS version of TestTrash. I think there's an issue with the FileSystem.listStatus methods between LocalFileSystem and DistributedFileSystem, which I'll look into. In the meantime, modified test so that it doesn't rely on that method and works on both local and distributed file systems.
          Will run full test suite tonight, report tomorrow morning. Also, deleted old patches to avoid confusion. New patches for both trunk and v20 should be good to go.

          Show
          Jakob Homan added a comment - Ran into a problem that I didn't notice with the HDFS version of TestTrash. I think there's an issue with the FileSystem.listStatus methods between LocalFileSystem and DistributedFileSystem, which I'll look into. In the meantime, modified test so that it doesn't rely on that method and works on both local and distributed file systems. Will run full test suite tonight, report tomorrow morning. Also, deleted old patches to avoid confusion. New patches for both trunk and v20 should be good to go.
          Jakob Homan made changes -
          Attachment HADOOP-6080-v20.patch [ 12412140 ]
          Attachment HADOOP-6080.patch [ 12412141 ]
          Hide
          Jakob Homan added a comment -

          Updated patches are good to go on all commons unit tests for trunk and all tests for v20. Test-patch is fine except the incorrect javac warnings, which are not related.

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          Show
          Jakob Homan added a comment - Updated patches are good to go on all commons unit tests for trunk and all tests for v20. Test-patch is fine except the incorrect javac warnings, which are not related. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Sanjay Radia added a comment -

          Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

          Show
          Sanjay Radia added a comment - Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?
          Hide
          Jakob Homan added a comment -

          Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

          I'm not a fan of special cases for certain directories, even for /tmp, and particularly when we're already straying away from the posix world with the trash feature. Minimizing surprise seems a good goal, and I'd be very surprised if I were accustomed to explicitly skipping the trash when I want and discovering something I had expected to be trashed had been helpfully nuked by the system.

          Show
          Jakob Homan added a comment - Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)? I'm not a fan of special cases for certain directories, even for /tmp, and particularly when we're already straying away from the posix world with the trash feature. Minimizing surprise seems a good goal, and I'd be very surprised if I were accustomed to explicitly skipping the trash when I want and discovering something I had expected to be trashed had been helpfully nuked by the system.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).

          The patch does not seem to have so many warnings. Filed HADOOP-6122.

          Show
          Tsz Wo Nicholas Sze added a comment - > [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). The patch does not seem to have so many warnings. Filed HADOOP-6122 .
          Hide
          Konstantin Shvachko added a comment -

          I just committed this. Thank you Jakob.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Jakob.
          Konstantin Shvachko made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.20.1 [ 12313866 ]
          Resolution Fixed [ 1 ]
          Hide
          Jakob Homan added a comment -

          added release note.

          Show
          Jakob Homan added a comment - added release note.
          Jakob Homan made changes -
          Release Note Provide a new option to rm and rmr, -skipTrash, which will immediately delete the files specified, rather than moving them to the trash.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/)
          is going to 0.20.
          . Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/ ) is going to 0.20. . Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.
          Eli Collins made changes -
          Link This issue is related to HADOOP-8598 [ HADOOP-8598 ]

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development