Hadoop Common
  1. Hadoop Common
  2. HADOOP-6203

Improve error message when moving to trash fails due to quota issue

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Improved error message suggests using -skpTrash option when hdfs -rm fails to move to trash because of quota.

      Description

      HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:

      $ hdfs -rmr /foo/bar/bat/boo
      
      rmr: Failed to move to trash:
      hdfs://cluster/foo/bar/bat/boo

      In this situation, the error message should say there was a quote problem and suggest -skipTrash.

      1. HADOOP-6203.patch
        2 kB
        Boris Shkolnik
      2. HADOOP-6203.patch
        1 kB
        Boris Shkolnik
      3. HADOOP-6203-1.patch
        0.6 kB
        Boris Shkolnik
      4. HADOOP-6203-1.patch
        2 kB
        Boris Shkolnik
      5. c6203_20091116_0.20.patch
        2 kB
        Tsz Wo Nicholas Sze

        Activity

        Hide
        Boris Shkolnik added a comment -

        Well, there are few ways of solving this.
        1. print out the "cause" exception together with the error message. That will tell the person that the problem was with Quota. We cannot make suggestions about "-skipTrash" in this case.
        2. catch and rethrow exception in delete function when trying to move to Trash. This way we know that the problem is with trash and we can issue a message suggesting using "-skipTrash". The disadvantage is that this way we issue two separate messages.

        Show
        Boris Shkolnik added a comment - Well, there are few ways of solving this. 1. print out the "cause" exception together with the error message. That will tell the person that the problem was with Quota. We cannot make suggestions about "-skipTrash" in this case. 2. catch and rethrow exception in delete function when trying to move to Trash. This way we know that the problem is with trash and we can issue a message suggesting using "-skipTrash". The disadvantage is that this way we issue two separate messages.
        Hide
        Jakob Homan added a comment -

        2 is probably preferable. The real implementation problem is that the Quota exception is in HDFS and so can't be caught and identified in Common.

        Show
        Jakob Homan added a comment - 2 is probably preferable. The real implementation problem is that the Quota exception is in HDFS and so can't be caught and identified in Common.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418014/HADOOP-6203.patch
        against trunk revision 808415.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418014/HADOOP-6203.patch against trunk revision 808415. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment -

        Currently there is no CLI facility in hadoop-common, so I resorted to manual testing:

        >bin/hdfs dfs -lsr /
        rw-rr- 1 0 2009-08-28 16:36 /file2
        rw-rr- 1 0 2009-08-28 16:37 /file3
        drwxr-xr-x - 0 2009-08-28 16:37 /user
        drwxr-xr-x - 0 2009-08-28 16:37 /user/borya
        drwxr-xr-x - 0 2009-08-28 16:37 /user/borya/.Trash
        drwx------ - 0 2009-08-28 16:37 /user/borya/.Trash/Current
        rw-rr- 1 0 2009-08-28 16:36 /user/borya/.Trash/Current/file1

        >bin/hdfs dfs -count -q /
        3 -3 none inf 4 2 0 hdfs://localhost:9000/

        >bin/hdfs dfs -rm /file3
        Problem with Trash. Consider using -skipTrash option <<<<<<<<< THIS IS the new line
        rm: Failed to move to trash: hdfs://localhost:9000/file3

        Show
        Boris Shkolnik added a comment - Currently there is no CLI facility in hadoop-common, so I resorted to manual testing: >bin/hdfs dfs -lsr / rw-r r - 1 0 2009-08-28 16:36 /file2 rw-r r - 1 0 2009-08-28 16:37 /file3 drwxr-xr-x - 0 2009-08-28 16:37 /user drwxr-xr-x - 0 2009-08-28 16:37 /user/borya drwxr-xr-x - 0 2009-08-28 16:37 /user/borya/.Trash drwx------ - 0 2009-08-28 16:37 /user/borya/.Trash/Current rw-r r - 1 0 2009-08-28 16:36 /user/borya/.Trash/Current/file1 >bin/hdfs dfs -count -q / 3 -3 none inf 4 2 0 hdfs://localhost:9000/ >bin/hdfs dfs -rm /file3 Problem with Trash. Consider using -skipTrash option <<<<<<<<< THIS IS the new line rm: Failed to move to trash: hdfs://localhost:9000/file3
        Hide
        Suresh Srinivas added a comment -
        1. The bug description talks about indicating that there was quota related problem while performing rm. That still is not captured with the new change. Also since the existing code just captures IOException from trashTmp.moveToTrash(), it not setup to capture quota issue.
        2. FSShell.delete() - This probably exists in the previous version of the code also. What happens if trashTmp.moveToTrash() returns -1 when skipTrash flag is true?
        Show
        Suresh Srinivas added a comment - The bug description talks about indicating that there was quota related problem while performing rm. That still is not captured with the new change. Also since the existing code just captures IOException from trashTmp.moveToTrash() , it not setup to capture quota issue. FSShell.delete() - This probably exists in the previous version of the code also. What happens if trashTmp.moveToTrash() returns -1 when skipTrash flag is true?
        Hide
        Boris Shkolnik added a comment -

        1. The idea is to make sure they know they can use skipTrash option. They can find real reason by looking at the logs. We can print exception.cause.message also, but that will make the message rather long. It will look like this:
        >bin/hdfs dfs -rm /file3
        Problem with Trash.org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory is exceeded: quota=3 file count=7. Consider using -skipTrash option
        rm: Failed to move to trash: hdfs://localhost:9000/file3

        We cannot catch Quota exception directly because it is part of hadoop-hdfs. Parsing string to find out that it is quota related doesn't sound like a good idea.

        Please specify if you like the longer version of the message.

        2. if skipTrash is true - moveToTrash() won't be called.

        Show
        Boris Shkolnik added a comment - 1. The idea is to make sure they know they can use skipTrash option. They can find real reason by looking at the logs. We can print exception.cause.message also, but that will make the message rather long. It will look like this: >bin/hdfs dfs -rm /file3 Problem with Trash.org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory is exceeded: quota=3 file count=7. Consider using -skipTrash option rm: Failed to move to trash: hdfs://localhost:9000/file3 We cannot catch Quota exception directly because it is part of hadoop-hdfs. Parsing string to find out that it is quota related doesn't sound like a good idea. Please specify if you like the longer version of the message. 2. if skipTrash is true - moveToTrash() won't be called.
        Hide
        Suresh Srinivas added a comment -

        I prefer the shorter message from the existing patch as output instead of printing the exception message in the output. Given that the layer where error is being printed does not have specific nature of error, such as, quota exceeded this seems like the best thing to do.

        Looking at the output which says {{rm: Failed to move to trash: hdfs://localhost:9000/file3 }}, the existing output (prior to the patch) already indicates a trash related error. The value add of this patch is that the output indicates the user can work around the problem by using the -skipTrash option.

        I am +1 for the change if Koji agrees that the existing output captures sufficient information for the user.

        Show
        Suresh Srinivas added a comment - I prefer the shorter message from the existing patch as output instead of printing the exception message in the output. Given that the layer where error is being printed does not have specific nature of error, such as, quota exceeded this seems like the best thing to do. Looking at the output which says {{rm: Failed to move to trash: hdfs://localhost:9000/file3 }}, the existing output (prior to the patch) already indicates a trash related error. The value add of this patch is that the output indicates the user can work around the problem by using the -skipTrash option. I am +1 for the change if Koji agrees that the existing output captures sufficient information for the user.
        Hide
        Boris Shkolnik added a comment -

        To me, as long as 'quota' error is shown, I'm happy.

        Koji

        Switching to the longer version.

        Show
        Boris Shkolnik added a comment - To me, as long as 'quota' error is shown, I'm happy. Koji Switching to the longer version.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418523/HADOOP-6203-1.patch
        against trunk revision 810756.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/13/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418523/HADOOP-6203-1.patch against trunk revision 810756. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/13/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12418525/HADOOP-6203-1.patch
        against trunk revision 810756.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418525/HADOOP-6203-1.patch against trunk revision 810756. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/15/console This message is automatically generated.
        Hide
        Boris Shkolnik added a comment -

        There is currently no CLI testing in hadoop-common.
        See my comment (few comments up) with results of manual testing.

        Show
        Boris Shkolnik added a comment - There is currently no CLI testing in hadoop-common. See my comment (few comments up) with results of manual testing.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch

        Show
        Suresh Srinivas added a comment - +1 for the patch
        Hide
        Suresh Srinivas added a comment -

        I committed this change. Thank you Boris.

        Show
        Suresh Srinivas added a comment - I committed this change. Thank you Boris.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #18 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/18/)
        . FsShell rm/rmr error message indicates exceeding Trash quota and suggests using -skpTrash, when moving to trash fails. Contributed by Boris Shkolnik.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #18 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/18/ ) . FsShell rm/rmr error message indicates exceeding Trash quota and suggests using -skpTrash, when moving to trash fails. Contributed by Boris Shkolnik.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/)
        . FsShell rm/rmr error message indicates exceeding Trash quota and suggests using -skpTrash, when moving to trash fails. Contributed by Boris Shkolnik.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #89 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/89/ ) . FsShell rm/rmr error message indicates exceeding Trash quota and suggests using -skpTrash, when moving to trash fails. Contributed by Boris Shkolnik.
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        c6203_20091116_0.20.patch: back ported HADOOP-6203-1.patch to 0.20

        All tests passed in 0.20, except TestDatanodeBlockScanner, TestFsck and TestReduceFetch, which are known issues.

        Show
        Tsz Wo Nicholas Sze added a comment - c6203_20091116_0.20.patch: back ported HADOOP-6203 -1.patch to 0.20 All tests passed in 0.20, except TestDatanodeBlockScanner, TestFsck and TestReduceFetch, which are known issues.

          People

          • Assignee:
            Boris Shkolnik
            Reporter:
            Jakob Homan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development