Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5939

FSDownload leaks FileSystem resources

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.5.1, 2.7.3
    • Fix Version/s: None
    • Component/s: nodemanager
    • Labels:
    • Flags:
      Important

      Description

      Background
      To use our self-defined FileSystem class, the item of configuration "fs.%s.impl.disable.cache" should set to true.

      In YARN's source code, the class named "org.apache.hadoop.yarn.util.FSDownload" use getFileSystem but never close, which leading to file descriptor leak because our self-defined FileSystem class close the file descriptor when the close function is invoked.

      My Question below:
      1. whether invoking "getFileSystem" but never close is YARN's expected behavior
      2. what should we do in our self-defined FileSystem resolve it.

        Attachments

        1. YARN-5939.03.patch
          14 kB
          Weiwei Yang
        2. YARN-5939.02.patch
          14 kB
          Weiwei Yang
        3. YARN-5939.01.patch
          14 kB
          Weiwei Yang
        4. YARN-5939.005.patch
          10 kB
          Weiwei Yang
        5. YARN-5939.004.patch
          10 kB
          Antal Bálint Steinbach

          Issue Links

            Activity

              People

              • Assignee:
                cheersyang Weiwei Yang
                Reporter:
                lxw668899 liuxiangwei
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified