Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6843

Entries in the FileSystem's Cache could be cleared when they are not used

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 0.20.0
    • None
    • None
    • None

    Description

      In FileSystem, there is a cache maintained for Filesystem instances. The entries in the cache are cleared only when explicit FileSystem.close is invoked. Applications are not careful on this aspect. Typically, they do FileSystem.get(), operate on the FileSystem, and then they just forget about it. Every FileSystem instance stores a reference to the Configuration object that it was created with. Over a period of time, as the cache grows, this can lead to OOM (we have seen this happening in our hadoop 20S clusters at Yahoo).

      This jira aims at addressing the above issue.

      Attachments

        1. fs-weak-ref.4.patch
          6 kB
          Devaraj Das
        2. TestFileSystemCache.java
          4 kB
          Tsz-wo Sze

        Activity

          People

            Unassigned Unassigned
            ddas Devaraj Das
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: