Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7094

LocalDistributedCacheManager leaves classloaders open, which leaks FDs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When a user starts a local mapred task from Hive's beeline, it will leave open file descriptors on the HS2 process (which runs the mapred task).

      I debugged this and saw that it is caused by LocalDistributedCacheManager class, which creates a new URLClassLoader, with a classpath for the two jars seen below. Somewhere down the line Loaders will be created in this URLClassLoader for these files effectively creating the FD's on the OS level.

      This is never cleaned up after execution, although LocalDistributedCacheManager removes the files, it will not close the ClassLoader, so FDs are left open although they point to deleted files at that time:

      [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
      java    14439 hive  DEL       REG                8,1             3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
      java    14439 hive  DEL       REG                8,1             3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
      java    14439 hive  649r      REG                8,1   8112438   3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar (deleted)
      java    14439 hive  650r      REG                8,1   8112438   3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
      
      

        Attachments

        1. MAPREDUCE-7094.6.patch
          5 kB
          Ádám Szita
        2. MAPREDUCE-7094.5.patch
          4 kB
          Ádám Szita
        3. MAPREDUCE-7094.4.patch
          4 kB
          Ádám Szita
        4. MAPREDUCE-7094.3.patch
          4 kB
          Ádám Szita
        5. MAPREDUCE-7094.2.patch
          2 kB
          Ádám Szita
        6. MAPREDUCE-7094.1.patch
          2 kB
          Ádám Szita
        7. MAPREDUCE-7094.0.patch
          2 kB
          Ádám Szita

          Activity

            People

            • Assignee:
              szita Ádám Szita
              Reporter:
              szita Ádám Szita
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: