Hadoop Common
  1. Hadoop Common
  2. HADOOP-8791

rm "Only deletes non empty directory and files."

    Details

    • Hadoop Flags:
      Reviewed

      Description

      The documentation (1.0.3) is describing the opposite of what rm does.
      It should be "Only delete files and empty directories."

      With regards to file, the size of the file should not matter, should it?

      OR I am totally misunderstanding the semantic of this command and I am not the only one.

      1. HADOOP-8791-trunk.patch
        1 kB
        Jing Zhao
      2. HADOOP-8791-trunk.patch
        1 kB
        Jing Zhao
      3. HADOOP-8791-trunk.001.patch
        2 kB
        Jing Zhao
      4. HADOOP-8791-branch-1.patch
        0.9 kB
        Jing Zhao
      5. HADOOP-8791-branch-1.patch
        1 kB
        Jing Zhao
      6. HADOOP-8791-branch-1.001.patch
        2 kB
        Jing Zhao

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/)
          svn merge -c 1390109 FIXES: HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1392437)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392437
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #392 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/392/ ) svn merge -c 1390109 FIXES: HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1392437) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1392437 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Robert Joseph Evans added a comment -

          I pulled this into branch-0.23

          Show
          Robert Joseph Evans added a comment - I pulled this into branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1208/)
          HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1208/ ) HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1177/)
          HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1177 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1177/ ) HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2791 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2791/)
          HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109)

          Result = FAILURE
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2791 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2791/ ) HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2832 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2832/)
          HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2832 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2832/ ) HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2769 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2769/)
          HADOOP-8791. Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109)

          Result = SUCCESS
          suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2769 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2769/ ) HADOOP-8791 . Fix rm command documentation to indicte it deletes files and not directories. Contributed by Jing Zhao. (Revision 1390109) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390109 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/file_system_shell.xml
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Jing for the patch. Thank you Bertrand for reporting the issue.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Jing for the patch. Thank you Bertrand for reporting the issue.
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch.

          Show
          Suresh Srinivas added a comment - +1 for the patch.
          Hide
          Jing Zhao added a comment -

          For trunk, it seems the rm can actually delete both files and directories (with -[rR]). The rm command also has new -f option. We should address all these new options in the document for trunk, and I will do that in HADOOP-8841.

          For branch-1, the rm only deletes files, so we can change the document based on the above discussion.

          Show
          Jing Zhao added a comment - For trunk, it seems the rm can actually delete both files and directories (with - [rR] ). The rm command also has new -f option. We should address all these new options in the document for trunk, and I will do that in HADOOP-8841 . For branch-1, the rm only deletes files, so we can change the document based on the above discussion.
          Hide
          Bertrand Dechoux added a comment -

          I have to confess that I am a bit lost.

          • currently (1.0.3) the rm delete only files (no directory, empty or not)
          • a rmdir command has been added to the trunk(cool)
          • rmr is still around in the trunk but a -r option was added to rm

          What is the difference between rmr and rm -r, then? And if there is such an option, I assume that the rm command will once again delete directories? It should be carefully documented.

          @Jing : You should add the -r option in the command description too for the trunk.
          => dfs -rm [-r] [-skipTrash]

          Show
          Bertrand Dechoux added a comment - I have to confess that I am a bit lost. currently (1.0.3) the rm delete only files (no directory, empty or not) a rmdir command has been added to the trunk(cool) rmr is still around in the trunk but a -r option was added to rm What is the difference between rmr and rm -r, then? And if there is such an option, I assume that the rm command will once again delete directories? It should be carefully documented. @Jing : You should add the -r option in the command description too for the trunk. => dfs -rm [-r] [-skipTrash]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546547/HADOOP-8791-trunk.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1518//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1518//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546547/HADOOP-8791-trunk.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1518//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1518//console This message is automatically generated.
          Hide
          Jing Zhao added a comment -

          Thanks for the comments Suresh! Will create the jira for rm document in trunk.

          Show
          Jing Zhao added a comment - Thanks for the comments Suresh! Will create the jira for rm document in trunk.
          Hide
          Suresh Srinivas added a comment -

          Patch looks good. Minor comment - under rmr we should mention it recursively deletes the directory and any content under it.

          @Jing can you please create another jira for the following: In trunk for command rm, the flags -[rR] and -f are not documented.

          Show
          Suresh Srinivas added a comment - Patch looks good. Minor comment - under rmr we should mention it recursively deletes the directory and any content under it. @Jing can you please create another jira for the following: In trunk for command rm, the flags - [rR] and -f are not documented.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12545268/HADOOP-8791-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545268/HADOOP-8791-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1469//console This message is automatically generated.
          Hide
          Jing Zhao added a comment -

          Updated for both branch-1 and trunk. Also mentioned "-rm -r" for the trunk patch.

          Show
          Jing Zhao added a comment - Updated for both branch-1 and trunk. Also mentioned "-rm -r" for the trunk patch.
          Hide
          Harsh J added a comment -

          Thanks Andy, missed that addition. We should just fix the docs then, on this issue.

          Show
          Harsh J added a comment - Thanks Andy, missed that addition. We should just fix the docs then, on this issue.
          Hide
          Andy Isaacson added a comment -

          we could add a new rmdir in FsShell

          There's already hdfs dfs -rmdir in trunk. It behaves like Unix rmdir: it only deletes empty directories, failing with "Directory is not empty" if there are files or subdirectories.

          Show
          Andy Isaacson added a comment - we could add a new rmdir in FsShell There's already hdfs dfs -rmdir in trunk. It behaves like Unix rmdir: it only deletes empty directories, failing with "Directory is not empty" if there are files or subdirectories.
          Hide
          Harsh J added a comment -

          That doc is pretty old so we may have regressed long ago as well. I suppose we could add a new rmdir in FsShell, as a client-side tool, instead of having rm delete directories - to stick to usual convention.

          I'd like Daryn's thoughts for a new command though.

          Show
          Harsh J added a comment - That doc is pretty old so we may have regressed long ago as well. I suppose we could add a new rmdir in FsShell, as a client-side tool, instead of having rm delete directories - to stick to usual convention. I'd like Daryn's thoughts for a new command though.
          Hide
          Hemanth Yamijala added a comment -

          The behaviour is the same with 1.0.2 as well.

          Show
          Hemanth Yamijala added a comment - The behaviour is the same with 1.0.2 as well.
          Hide
          Bertrand Dechoux added a comment -

          I made the same test on the 1.0.3 version and I can confirm that directories can not be delete, being empty or not has no influence...

          @Hemanth : so yes, the documentation should be updated to say only files.

          Can someone check for previous versions? Is that a regression? Or a documentation that was never correct? If that's a regression, should it be corrected now that's the 1.0.3 (and I guess the 1.0) behaviour?

          I also tested on 1.0.3 whether size of files has an impact but it doesn't.

          Second question : if the observed behaviour is the 'correct' one, it would really beg for a 'rmdir' equivalent command for HDFS.

          Show
          Bertrand Dechoux added a comment - I made the same test on the 1.0.3 version and I can confirm that directories can not be delete, being empty or not has no influence... @Hemanth : so yes, the documentation should be updated to say only files. Can someone check for previous versions? Is that a regression? Or a documentation that was never correct? If that's a regression, should it be corrected now that's the 1.0.3 (and I guess the 1.0) behaviour? I also tested on 1.0.3 whether size of files has an impact but it doesn't. Second question : if the observed behaviour is the 'correct' one, it would really beg for a 'rmdir' equivalent command for HDFS.
          Hide
          Hemanth Yamijala added a comment -

          Also, I think the examples in the same documentation section might need update to reflect that empty directories can't be removed.

          Show
          Hemanth Yamijala added a comment - Also, I think the examples in the same documentation section might need update to reflect that empty directories can't be removed.
          Hide
          Hemanth Yamijala added a comment -

          It looks like rm cannot even delete empty directories. Tried this on both 1.0.3 and trunk. We should modify the documentation to only specify that it deletes files, right ?

          Show
          Hemanth Yamijala added a comment - It looks like rm cannot even delete empty directories. Tried this on both 1.0.3 and trunk. We should modify the documentation to only specify that it deletes files, right ?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544860/HADOOP-8791-trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1443//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1443//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544860/HADOOP-8791-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1443//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1443//console This message is automatically generated.
          Hide
          Jing Zhao added a comment -

          So I did the two patches for branch-1 and trunk for Bertrand. Thanks for the finding Bertrand!

          Show
          Jing Zhao added a comment - So I did the two patches for branch-1 and trunk for Bertrand. Thanks for the finding Bertrand!
          Hide
          Bertrand Dechoux added a comment -

          And "non empty directories" is actually "empty directories". This one is not ambiguous but wrong.

          I might send a patch but that won't be soon. I might have a go at it around 22-23th but no promise.

          Show
          Bertrand Dechoux added a comment - And "non empty directories" is actually "empty directories". This one is not ambiguous but wrong. I might send a patch but that won't be soon. I might have a go at it around 22-23th but no promise.
          Hide
          Harsh J added a comment -

          The original doc under question is:

          Delete files specified as args. Only deletes non empty directory and files.

          While there's actually supposed to be a separation, i.e. "non empty directories" AND "files" in reading this, you're right that this is ambiguous.

          Wanna send across a patch for both branch-1 and trunk Bertrand?

          Show
          Harsh J added a comment - The original doc under question is: Delete files specified as args. Only deletes non empty directory and files. While there's actually supposed to be a separation, i.e. "non empty directories" AND "files" in reading this, you're right that this is ambiguous. Wanna send across a patch for both branch-1 and trunk Bertrand?

            People

            • Assignee:
              Jing Zhao
              Reporter:
              Bertrand Dechoux
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development