Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3098

Memory leak from large number of FileSystem instances in FileSystem.CACHE

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.9.0
    • 0.9.1, 0.10.0
    • Shims
    • None

    • Running with Hadoop 20.205.0.3+ / 1.0.x with security turned on.

    Description

      The problem manifested from stress-testing HCatalog 0.4.1 (as part of testing the Oracle backend).

      The HCatalog server ran out of memory (-Xmx2048m) when pounded by 60-threads, in under 24 hours. The heap-dump indicates that hadoop::FileSystem.CACHE had 1000000 instances of FileSystem, whose combined retained-mem consumed the entire heap.

      It boiled down to hadoop::UserGroupInformation::equals() being implemented such that the "Subject" member is compared for equality ("=="), and not equivalence (".equals()"). This causes equivalent UGI instances to compare as unequal, and causes a new FileSystem instance to be created and cached.

      The UGI.equals() is so implemented, incidentally, as a fix for yet another problem (HADOOP-6670); so it is unlikely that that implementation can be modified.

      The solution for this is to check for UGI equivalence in HCatalog (i.e. in the Hive metastore), using an cache for UGI instances in the shims.

      I have a patch to fix this. I'll upload it shortly. I just ran an overnight test to confirm that the memory-leak has been arrested.

      Attachments

        1. Hive_3098.patch
          7 kB
          Mithun Radhakrishnan
        2. Hive-3098_(FS_closeAllForUGI()).patch
          2 kB
          Mithun Radhakrishnan
        3. hive-3098.patch
          5 kB
          Mithun Radhakrishnan

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HIVE-9234 HiveServer2 leaks FileSystem objects in FileSystem.CACHE

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-3545 DFSClient leak due to malfunctioning of FileSystem Cache

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HIVE-4501 HS2 memory leak - FileSystem objects in FileSystem.CACHE

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HIVE-5296 Memory leak: OOM Error after multiple open/closed JDBC connections.

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-17214 Allow file system caching to be disabled for all file systems

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-3513 HttpFS should cache filesystems

          • Major - Major loss of function.
          • Closed

          Activity

            People

              mithun Mithun Radhakrishnan
              mithun Mithun Radhakrishnan
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: