Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14600

LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 3.1.0
    • Component/s: fs
    • Labels:
      None
    • Environment:

      file:// in a dir with many files

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Reported in SPARK-21137. a FileSystem.listStatus call really craws against the local FS, because FileStatus.getPemissions call forces DeprecatedRawLocalFileStatus tp spawn a process to read the real UGI values.

      That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's a process exec/spawn, with all the costs. This gets expensive if you have many files.

        Attachments

        1. HADOOP-14600.001.patch
          8 kB
          Ping Liu
        2. TestRawLocalFileSystemContract.java
          3 kB
          Ping Liu
        3. HADOOP-14600.002.patch
          17 kB
          Ping Liu
        4. HADOOP-14600.003.patch
          28 kB
          Ping Liu
        5. HADOOP-14600.004.patch
          28 kB
          Ping Liu
        6. HADOOP-14600.005.patch
          29 kB
          Ping Liu
        7. HADOOP-14600.006.patch
          29 kB
          Ping Liu
        8. HADOOP-14600.007.patch
          29 kB
          Christopher Douglas
        9. HADOOP-14600.008.patch
          28 kB
          Christopher Douglas
        10. HADOOP-14600.009.patch
          28 kB
          Christopher Douglas
        11. command_line_test_result__linux.txt
          107 kB
          Ping Liu
        12. command_line_test_result__windows.txt
          182 kB
          Ping Liu

          Issue Links

            Activity

              People

              • Assignee:
                myapachejira Ping Liu
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: