Handling of Trash with quota

• Key: HADOOP-6080
• Type: New Feature
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Jakob Homan
• Reporter: Koji Noguchi
• Component/s: fs
• Fix Version/s: 0.20.1

Description

Currently with quota turned on, user cannot call '-rmr' on large directory that causes over quota.

[knoguchi src]$ hadoop dfs -rmr /tmp/net2
rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
[knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/knoguchi is exceeded: namespace
quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353

Besides from error message being unfriendly, how should this be handled?

Comments

[Koji Noguchi]

1) Tell users to use -Dfs.trash.interval=0 when deleting large directory
2) Exclude /user/<username>/.Trash from the quota
3) Move .Trash out of /user directory. Maybe /Trash/<username> and set different quota.
4) When -rm/rmr fail with quota, automatically delete them.
5) Introduce a separate command that does (1). Something like -rmr -skipTrash for force delete.

[Raghu Angadi]

+1 for (5).

[Tsz Wo (Nicholas), SZE]

If we do (5), we still have to do (1), i.e. add a message telling the user to use -skipTrash .

[Jakob Homan]

I'm going to go ahead and implement 5.

[Jakob Homan]

Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which performs as expected. Adds to trash unit test to verify correct execution. Changes documentation to reflect new option. Docs suggest this option as being a solution when a directory is over quota.

Passes all commons unit tests. Running test patch now. Will post those results when done.

[Jakob Homan]

submitting patch.

[Tsz Wo (Nicholas), SZE]

+1 patch looks good

[Jakob Homan]

Test-patch:

[exec] -1 overall.  
[exec] 
[exec]     +1 @author.  The patch does not contain any @author tags.
[exec] 
[exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
[exec] 
[exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
[exec] 
[exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
[exec] 
[exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
[exec] 
[exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

This is weird. I've attached the javac warnings it says are new and they have nothing to do with this patch. test-patch must be broken in this regard. I believe the patch is ready to go.

[Jakob Homan]

Attaching two new files:

• Updated patch. Previous patch missed updating the help text for rmr to include -skipTrash option. No change to actual code.
• Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to be changed for patch, just file locations were different. Code is still the same. Passes unit tests.

[Jakob Homan]

Canceling patch to double check something.

[Jakob Homan]

Ran into a problem that I didn't notice with the HDFS version of TestTrash. I think there's an issue with the FileSystem.listStatus methods between LocalFileSystem and DistributedFileSystem, which I'll look into. In the meantime, modified test so that it doesn't rely on that method and works on both local and distributed file systems.
Will run full test suite tonight, report tomorrow morning. Also, deleted old patches to avoid confusion. New patches for both trunk and v20 should be good to go.

[Jakob Homan]

Updated patches are good to go on all commons unit tests for trunk and all tests for v20. Test-patch is fine except the incorrect javac warnings, which are not related.

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

[Sanjay Radia]

Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

[Jakob Homan]

Should we consider excluding trash for files deleted from /tmp (ie make -skipTrash implicit when deleting from /tmp.)?

I'm not a fan of special cases for certain directories, even for /tmp, and particularly when we're already straying away from the posix world with the trash feature. Minimizing surprise seems a good goal, and I'd be very surprised if I were accustomed to explicitly skipping the trash when I want and discovering something I had expected to be trashed had been helpfully nuked by the system.

[Tsz Wo (Nicholas), SZE]

> [exec] -1 javac. The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).

The patch does not seem to have so many warnings. Filed HADOOP-6122.

[Konstantin Shvachko]

I just committed this. Thank you Jakob.

[Jakob Homan]

added release note.

[Hudson]

Integrated in Hadoop-Common-trunk #13 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/)
is going to 0.20.
. Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.