Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16378

RawLocalFileStatus throws exception if a file is created and deleted quickly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.3.0
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None
    • Environment:

      Ubuntu 18.04, Hadoop 2.7.3 (Though this problem exists on later versions of Hadoop as well), Java 8 ( + Java 11).

    • Flags:
      Patch

      Description

      Bug occurs when NFS creates temporary ".nfs*" files as part of file moves and accesses. If this file is deleted very quickly after being created, a RuntimeException is thrown. The root cause is in the loadPermissionInfo method in org.apache.hadoop.fs.RawLocalFileSystem. To get the permission info, it first does

       

      ls -ld

       and then attempts to get permissions info about each file. If a file disappears between these two steps, an exception is thrown.

      Reproduction Steps:

      An isolated way to reproduce the bug is to run FileInputFormat.listStatus over and over on the same dir that we’re creating those temp files in. On Ubuntu or any other Linux-based system, this should fail intermittently

      Fix:

      One way in which we managed to fix this was to ignore the exception being thrown in loadPemissionInfo() if the exit code is 1 or 2. Alternatively, it's possible that turning "useDeprecatedFileStatus" off in RawLocalFileSystem would fix this issue, though we never tested this, and the flag was implemented to fix HADOOP-9652. Could also fix in conjunction with HADOOP-8772.

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              anon858 K S
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: