Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8772

RawLocalFileStatus shells out for permission info

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 0.23.3, 2.0.2-alpha, 3.0.0-alpha1
    • None
    • fs
    • None

    Description

      RawLocalFileStatus shells out to run "ls" to get permissions info. This very inefficient.

      More importantly, mixing mulithreading and forking is risky. Some version of glibc in RHEL will deadlock things such as "_GI_IO_list_lock" and "malloc_atfork". All this unnecessary shelling out to access the local filesystem greatly increases the risk of deadlock.

      Namely, the NM's user localizer is seen to jam more frequently than the TT & NM.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17839 LocalFS to support ability to disable permission get/set; remove need for winutils

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              daryn Daryn Sharp
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: