Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.20.1
    • fs
    • None
    • Reviewed
    • Provide a new option to rm and rmr, -skipTrash, which will immediately delete the files specified, rather than moving them to the trash.

    Description

      Currently with quota turned on, user cannot call '-rmr' on large directory that causes over quota.

      [knoguchi src]$ hadoop dfs -rmr /tmp/net2
      rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
      [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
      mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/knoguchi is exceeded: namespace
      quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
      

      Besides from error message being unfriendly, how should this be handled?

      Attachments

        1. javac_warnings_diff.txt
          17 kB
          Jakob Homan
        2. HADOOP-6080-v20.patch
          9 kB
          Jakob Homan
        3. HADOOP-6080.patch
          9 kB
          Jakob Homan

        Issue Links

          Activity

            knoguchi Koji Noguchi added a comment -

            1) Tell users to use -Dfs.trash.interval=0 when deleting large directory
            2) Exclude /user/<username>/.Trash from the quota
            3) Move .Trash out of /user directory. Maybe /Trash/<username> and set different quota.
            4) When -rm/rmr fail with quota, automatically delete them.
            5) Introduce a separate command that does (1). Something like -rmr -skipTrash for force delete.

            knoguchi Koji Noguchi added a comment - 1) Tell users to use -Dfs.trash.interval=0 when deleting large directory 2) Exclude /user/<username>/.Trash from the quota 3) Move .Trash out of /user directory. Maybe /Trash/<username> and set different quota. 4) When -rm/rmr fail with quota, automatically delete them. 5) Introduce a separate command that does (1). Something like -rmr -skipTrash for force delete.
            rangadi Raghu Angadi added a comment -

            +1 for (5).

            rangadi Raghu Angadi added a comment - +1 for (5).
            szetszwo Tsz-wo Sze added a comment -

            If we do (5), we still have to do (1), i.e. add a message telling the user to use -skipTrash .

            szetszwo Tsz-wo Sze added a comment - If we do (5), we still have to do (1), i.e. add a message telling the user to use -skipTrash .
            jghoman Jakob Homan added a comment -

            I'm going to go ahead and implement 5.

            jghoman Jakob Homan added a comment - I'm going to go ahead and implement 5.
            jghoman Jakob Homan added a comment -

            Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which performs as expected. Adds to trash unit test to verify correct execution. Changes documentation to reflect new option. Docs suggest this option as being a solution when a directory is over quota.

            Passes all commons unit tests. Running test patch now. Will post those results when done.

            jghoman Jakob Homan added a comment - Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which performs as expected. Adds to trash unit test to verify correct execution. Changes documentation to reflect new option. Docs suggest this option as being a solution when a directory is over quota. Passes all commons unit tests. Running test patch now. Will post those results when done.
            jghoman Jakob Homan added a comment -

            submitting patch.

            jghoman Jakob Homan added a comment - submitting patch.
            szetszwo Tsz-wo Sze added a comment -

            +1 patch looks good

            szetszwo Tsz-wo Sze added a comment - +1 patch looks good
            jghoman Jakob Homan added a comment -

            Test-patch:

            [exec] -1 overall.  
            [exec] 
            [exec]     +1 @author.  The patch does not contain any @author tags.
            [exec] 
            [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
            [exec] 
            [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
            [exec] 
            [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
            [exec] 
            [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
            [exec] 
            [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
            

            This is weird. I've attached the javac warnings it says are new and they have nothing to do with this patch. test-patch must be broken in this regard. I believe the patch is ready to go.

            jghoman Jakob Homan added a comment - Test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. This is weird. I've attached the javac warnings it says are new and they have nothing to do with this patch. test-patch must be broken in this regard. I believe the patch is ready to go.
            jghoman Jakob Homan added a comment -

            Attaching two new files:

            • Updated patch. Previous patch missed updating the help text for rmr to include -skipTrash option. No change to actual code.
            • Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to be changed for patch, just file locations were different. Code is still the same. Passes unit tests.
            jghoman Jakob Homan added a comment - Attaching two new files: Updated patch. Previous patch missed updating the help text for rmr to include -skipTrash option. No change to actual code. Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to be changed for patch, just file locations were different. Code is still the same. Passes unit tests.
            jghoman Jakob Homan added a comment -

            Canceling patch to double check something.

            jghoman Jakob Homan added a comment - Canceling patch to double check something.
            jghoman Jakob Homan added a comment -

            Ran into a problem that I didn't notice with the HDFS version of TestTrash. I think there's an issue with the FileSystem.listStatus methods between LocalFileSystem and DistributedFileSystem, which I'll look into. In the meantime, modified test so that it doesn't rely on that method and works on both local and distributed file systems.
            Will run full test suite tonight, report tomorrow morning. Also, deleted old patches to avoid confusion. New patches for both trunk and v20 should be good to go.

            jghoman Jakob Homan added a comment - Ran into a problem that I didn't notice with the HDFS version of TestTrash. I think there's an issue with the FileSystem.listStatus methods between LocalFileSystem and DistributedFileSystem, which I'll look into. In the meantime, modified test so that it doesn't rely on that method and works on both local and distributed file systems. Will run full test suite tonight, report tomorrow morning. Also, deleted old patches to avoid confusion. New patches for both trunk and v20 should be good to go.
            jghoman Jakob Homan added a comment -

            Updated patches are good to go on all commons unit tests for trunk and all tests for v20. Test-patch is fine except the incorrect javac warnings, which are not related.

                 [exec] -1 overall.  
                 [exec] 
                 [exec]     +1 @author.  The patch does not contain any @author tags.
                 [exec] 
                 [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
                 [exec] 
                 [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
                 [exec] 
                 [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
                 [exec] 
                 [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
                 [exec] 
                 [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
            jghoman Jakob Homan added a comment - Updated patches are good to go on all commons unit tests for trunk and all tests for v20. Test-patch is fine except the incorrect javac warnings, which are not related. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
            sanjay.radia Sanjay Radia added a comment -

            Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

            sanjay.radia Sanjay Radia added a comment - Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?
            jghoman Jakob Homan added a comment -

            Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

            I'm not a fan of special cases for certain directories, even for /tmp, and particularly when we're already straying away from the posix world with the trash feature. Minimizing surprise seems a good goal, and I'd be very surprised if I were accustomed to explicitly skipping the trash when I want and discovering something I had expected to be trashed had been helpfully nuked by the system.

            jghoman Jakob Homan added a comment - Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)? I'm not a fan of special cases for certain directories, even for /tmp, and particularly when we're already straying away from the posix world with the trash feature. Minimizing surprise seems a good goal, and I'd be very surprised if I were accustomed to explicitly skipping the trash when I want and discovering something I had expected to be trashed had been helpfully nuked by the system.
            szetszwo Tsz-wo Sze added a comment -

            > [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).

            The patch does not seem to have so many warnings. Filed HADOOP-6122.

            szetszwo Tsz-wo Sze added a comment - > [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings). The patch does not seem to have so many warnings. Filed HADOOP-6122 .

            I just committed this. Thank you Jakob.

            shv Konstantin Shvachko added a comment - I just committed this. Thank you Jakob.
            jghoman Jakob Homan added a comment -

            added release note.

            jghoman Jakob Homan added a comment - added release note.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/)
            is going to 0.20.
            . Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/ ) is going to 0.20. . Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.

            People

              jghoman Jakob Homan
              knoguchi Koji Noguchi
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: