Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7094

LocalDistributedCacheManager leaves classloaders open, which leaks FDs

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      When a user starts a local mapred task from Hive's beeline, it will leave open file descriptors on the HS2 process (which runs the mapred task).

      I debugged this and saw that it is caused by LocalDistributedCacheManager class, which creates a new URLClassLoader, with a classpath for the two jars seen below. Somewhere down the line Loaders will be created in this URLClassLoader for these files effectively creating the FD's on the OS level.

      This is never cleaned up after execution, although LocalDistributedCacheManager removes the files, it will not close the ClassLoader, so FDs are left open although they point to deleted files at that time:

      [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
      java    14439 hive  DEL       REG                8,1             3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
      java    14439 hive  DEL       REG                8,1             3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
      java    14439 hive  649r      REG                8,1   8112438   3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar (deleted)
      java    14439 hive  650r      REG                8,1   8112438   3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)


        1. MAPREDUCE-7094.0.patch
          2 kB
          Ádám Szita
        2. MAPREDUCE-7094.1.patch
          2 kB
          Ádám Szita
        3. MAPREDUCE-7094.2.patch
          2 kB
          Ádám Szita
        4. MAPREDUCE-7094.3.patch
          4 kB
          Ádám Szita
        5. MAPREDUCE-7094.4.patch
          4 kB
          Ádám Szita
        6. MAPREDUCE-7094.5.patch
          4 kB
          Ádám Szita
        7. MAPREDUCE-7094.6.patch
          5 kB
          Ádám Szita



            • Assignee:
              szita Ádám Szita
              szita Ádám Szita


              • Created:

                Issue deployment