Hadoop Common
  1. Hadoop Common
  2. HADOOP-6536

FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so.
      Or it should be changed not to delete the contents of the sym-linked directory.

      1. HADOOP-6536.patch
        3 kB
        Ravi Gummadi
      2. HADOOP-6536.v1.patch
        3 kB
        Ravi Gummadi
      3. HADOOP-6536.v1.1.patch
        5 kB
        Ravi Gummadi

        Activity

        Hide
        Ravi Gummadi added a comment -

        Since when symlink to a file is sent as param to fullyDelete(), it deletes the symlink only(and not the file pointed to by symlink), I think fullyDelete() should only delete the symlink even when symlink to a dir is passed as param(and should not delete the dir pointed to by symlink).

        Show
        Ravi Gummadi added a comment - Since when symlink to a file is sent as param to fullyDelete(), it deletes the symlink only(and not the file pointed to by symlink), I think fullyDelete() should only delete the symlink even when symlink to a dir is passed as param(and should not delete the dir pointed to by symlink).
        Hide
        Hemanth Yamijala added a comment -

        Based on a simple test I did on Linux, it appears that removal of a symlink (whether to file or directory) only removes the link. If I treat FileUtil.fullyDelete as a Java equivalent for "rm -r", then it seems to me that there's a bug in the implementation of FileUtil.fullyDelete. It should only delete the link and not contents of it. IOW, I am +1 for Ravi's proposal.

        Show
        Hemanth Yamijala added a comment - Based on a simple test I did on Linux, it appears that removal of a symlink (whether to file or directory) only removes the link. If I treat FileUtil.fullyDelete as a Java equivalent for "rm -r", then it seems to me that there's a bug in the implementation of FileUtil.fullyDelete. It should only delete the link and not contents of it. IOW, I am +1 for Ravi's proposal.
        Hide
        Eli Collins added a comment -

        +1 Ravi's proposal. I would also add that fullyDelete should delete dangling links (it currently does but we should add a test). The current behavior is due to java making it difficult to identify symlinks.

        Show
        Eli Collins added a comment - +1 Ravi's proposal. I would also add that fullyDelete should delete dangling links (it currently does but we should add a test). The current behavior is due to java making it difficult to identify symlinks.
        Hide
        Ravi Gummadi added a comment -

        One more issue.

        Currently FileUtil.fullyDelete(myDir) comes out stopping deletion of other files/directories if it is unable to delete a file/dir(say because of not having permissions to delete that file/dir) anywhere under myDir. This is because we return from method if the recursive call "if(!fullyDelete())

        {return false;}

        " fails at any level of recursion.

        Shouldn't it continue with deletion of other files/dirs continuing in the for loop instead of returning false here ? We can just set a boolean to false in this if() and can continue the for loop sothat return value of fullyDelete() is correct always.

        For example, if we have myDir/subdir1/file1, myDir/subdir2/file2 and if fullyDelete() tries to delete file1 first and could not delete it(because subDir1 has nonwritable permissions), then fullyDelete() currently comes out without deleting file2 and subdir2.

        I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf').

        Thoughts ?

        Show
        Ravi Gummadi added a comment - One more issue. Currently FileUtil.fullyDelete(myDir) comes out stopping deletion of other files/directories if it is unable to delete a file/dir(say because of not having permissions to delete that file/dir) anywhere under myDir. This is because we return from method if the recursive call "if(!fullyDelete()) {return false;} " fails at any level of recursion. Shouldn't it continue with deletion of other files/dirs continuing in the for loop instead of returning false here ? We can just set a boolean to false in this if() and can continue the for loop sothat return value of fullyDelete() is correct always. For example, if we have myDir/subdir1/file1, myDir/subdir2/file2 and if fullyDelete() tries to delete file1 first and could not delete it(because subDir1 has nonwritable permissions), then fullyDelete() currently comes out without deleting file2 and subdir2. I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf'). Thoughts ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf').

        +1 for this. But this is a separate issue, so I am creating a new ticket for this.

        Show
        Vinod Kumar Vavilapalli added a comment - I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf'). +1 for this. But this is a separate issue, so I am creating a new ticket for this.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf').

        Filed HADOOP-6631 for this.

        Show
        Vinod Kumar Vavilapalli added a comment - I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf'). Filed HADOOP-6631 for this.
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for trunk. This patch is on top of HADOOP-6631.

        Show
        Ravi Gummadi added a comment - Attaching patch for trunk. This patch is on top of HADOOP-6631 .
        Hide
        Ravi Gummadi added a comment -

        Attaching new patch to apply to the latest trunk as HADOOP-6631 has gone in.

        Show
        Ravi Gummadi added a comment - Attaching new patch to apply to the latest trunk as HADOOP-6631 has gone in.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Took a look at the latest patch, the changes seem fine.

        Show
        Sreekanth Ramakrishnan added a comment - Took a look at the latest patch, the changes seem fine.
        Hide
        Ravi Gummadi added a comment -

        Allowing the patch to go through Hudson...

        Show
        Ravi Gummadi added a comment - Allowing the patch to go through Hudson...
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12443923/HADOOP-6536.v1.patch
        against trunk revision 941662.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443923/HADOOP-6536.v1.patch against trunk revision 941662. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/510/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Changes look good.

        I would also add that fullyDelete should delete dangling links (it currently does but we should add a test).

        Can you add a test for deleting dangling links also?

        Show
        Amareshwari Sriramadasu added a comment - Changes look good. I would also add that fullyDelete should delete dangling links (it currently does but we should add a test). Can you add a test for deleting dangling links also?
        Hide
        Ravi Gummadi added a comment -

        Attaching patch adding more testcases.

        Show
        Ravi Gummadi added a comment - Attaching patch adding more testcases.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch looks good.

        Show
        Amareshwari Sriramadasu added a comment - Patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch
        against trunk revision 964993.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch against trunk revision 964993. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/console This message is automatically generated.
        Hide
        Ravi Gummadi added a comment -

        The failed test TestTrash passes on my local machine. Allowing to go through Hudson again...

        Show
        Ravi Gummadi added a comment - The failed test TestTrash passes on my local machine. Allowing to go through Hudson again...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch
        against trunk revision 965696.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch against trunk revision 965696. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/console This message is automatically generated.
        Hide
        Ravi Gummadi added a comment -

        >> -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
        No javadoc warning is introduced in this patch.

        Show
        Ravi Gummadi added a comment - >> -1 javadoc. The javadoc tool appears to have generated 1 warning messages. No javadoc warning is introduced in this patch.
        Hide
        Amareshwari Sriramadasu added a comment -

        I just committed this. Thanks Ravi!

        Show
        Amareshwari Sriramadasu added a comment - I just committed this. Thanks Ravi!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #332 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/332/)
        HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #332 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/332/ ) HADOOP-6536 . Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #398 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/)
        HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #398 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/ ) HADOOP-6536 . Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi

          People

          • Assignee:
            Ravi Gummadi
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development