Hadoop Common
  1. Hadoop Common
  2. HADOOP-8849

FileUtil#fullyDelete should grant the target directories +rwx permissions before trying to delete them

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.3-alpha, 0.23.6
    • Fix Version/s: 2.0.3-alpha, 0.23.7
    • Component/s: None
    • Labels:
      None

      Description

      2 improvements are suggested for implementation of methods org.apache.hadoop.fs.FileUtil.fullyDelete(File) and org.apache.hadoop.fs.FileUtil.fullyDeleteContents(File):

      1) We should grant +rwx permissions the target directories before trying to delete them.
      The mentioned methods fail to delete directories that don't have read or execute permissions.
      Actual problem appears if an hdfs-related test is timed out (with a short timeout like tens of seconds), and the forked test process is killed, some directories are left on disk that are not readable and/or executable. This prevents next tests from being executed properly because these directories cannot be deleted with FileUtil#fullyDelete(), so many subsequent tests fail. So, its recommended to grant the read, write, and execute permissions the directories whose content is to be deleted.

      2) Generic reliability improvement: we shouldn't rely upon File#delete() return value, use File#exists() instead.
      FileUtil#fullyDelete() uses return value of method java.io.File#delete(), but this is not reliable because File#delete() returns true only if the file was deleted as a result of the #delete() method invocation. E.g. in the following code
      if (f.exists())

      { // 1 return f.delete(); // 2 }

      if the file f was deleted by another thread or process between calls "1" and "2", this fragment will return "false", while the file f does not exist upon the method return.
      So, better to write
      if (f.exists())

      { f.delete(); return !f.exists(); }
      1. HADOOP-8849-trunk--5.patch
        13 kB
        Ivan A. Veselovsky
      2. HADOOP-8849-vs-trunk-4.patch
        7 kB
        Ivan A. Veselovsky

        Activity

        Ivan A. Veselovsky created issue -
        Ivan A. Veselovsky made changes -
        Field Original Value New Value
        Attachment HADOOP-8849-vs-trunk.patch [ 12546696 ]
        Ivan A. Veselovsky made changes -
        Description 2 improvements are suggested for implementation of methods org.apache.hadoop.fs.FileUtil.fullyDelete(File) and org.apache.hadoop.fs.FileUtil.fullyDeleteContents(File):
         
        1) We should grant +rwx permissions the target directories before trying to delete them.
        The mentioned methods fail to dlete directories that don't have read or execute permissions.
        Actual problem appears if an hdfs-related test is timed out (with a short timeout like tesns of seconds), and the forked test process is killed, some directories are left on disk that are not readable and/or executable. This prevents next tests from being executed properly because these directories cannot be deleted with FileUtil#fullyDelete(), so many subsequent tests fail. So, its recommended to grant the read, write, and execute permissions the directories whose content is to be deleted.

        2) We shouldn't rely upon File#delete() return value, use File#exists() instead.
        FileUtil#fullyDelete() uses return value of method java.io.File#delete(), but this is not reliable because File#delete() returns true only if the file was deleted as a result of the #delete() method invocation. E.g. in the following code
        if (f.exists()) { // 1
          return f.delete(); // 2
        }
        if the file f was deleted by another thread or process between calls "1" and "2", this fragment will return "false", while the file f does not exist upon the method return.
        So, better to write
        if (f.exists()) {
          f.delete();
          return !f.exists();
        }
        2 improvements are suggested for implementation of methods org.apache.hadoop.fs.FileUtil.fullyDelete(File) and org.apache.hadoop.fs.FileUtil.fullyDeleteContents(File):
         
        1) We should grant +rwx permissions the target directories before trying to delete them.
        The mentioned methods fail to delete directories that don't have read or execute permissions.
        Actual problem appears if an hdfs-related test is timed out (with a short timeout like tens of seconds), and the forked test process is killed, some directories are left on disk that are not readable and/or executable. This prevents next tests from being executed properly because these directories cannot be deleted with FileUtil#fullyDelete(), so many subsequent tests fail. So, its recommended to grant the read, write, and execute permissions the directories whose content is to be deleted.

        2) Generic reliability improvement: we shouldn't rely upon File#delete() return value, use File#exists() instead.
        FileUtil#fullyDelete() uses return value of method java.io.File#delete(), but this is not reliable because File#delete() returns true only if the file was deleted as a result of the #delete() method invocation. E.g. in the following code
        if (f.exists()) { // 1
          return f.delete(); // 2
        }
        if the file f was deleted by another thread or process between calls "1" and "2", this fragment will return "false", while the file f does not exist upon the method return.
        So, better to write
        if (f.exists()) {
          f.delete();
          return !f.exists();
        }
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk.patch [ 12546696 ]
        Ivan A. Veselovsky made changes -
        Assignee Ivan A. Veselovsky [ iveselovsky ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk-2.patch [ 12547980 ]
        Ivan A. Veselovsky made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk-2.patch [ 12547980 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk-3.patch [ 12547999 ]
        Ivan A. Veselovsky made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Ivan A. Veselovsky made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk-3.patch [ 12547999 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-vs-trunk-4.patch [ 12548002 ]
        Ivan A. Veselovsky made changes -
        Attachment HADOOP-8849-trunk--5.patch [ 12564973 ]
        Ivan A. Veselovsky made changes -
        Affects Version/s 3.0.0 [ 12320357 ]
        Affects Version/s 2.0.3-alpha [ 12323273 ]
        Affects Version/s 0.23.6 [ 12323504 ]
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 3.0.0 [ 12320357 ]
        Fix Version/s 2.0.3-alpha [ 12323273 ]
        Fix Version/s 0.23.7 [ 12323956 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Allen Wittenauer made changes -
        Fix Version/s 3.0.0 [ 12320357 ]

          People

          • Assignee:
            Ivan A. Veselovsky
            Reporter:
            Ivan A. Veselovsky
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development