Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9040

LevelDBCacheTimelineStore in ATS 1.5 leaks native memory

    Details

      Description

      When LevelDBCacheTimelineStore from YARN-4219 is used as ATS 1.5 entity caching storage, we observe memory leak due to leveldb files even after the fix of YARN-5368 .

      Top output shows 0.024TB (25GB) RES, even though heap size is only 8GB.

       

       

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      25519 yarn 20 0 33.024g 0.024t 41468 S 6.2 26.0 21:07.39 /usr/java/default/bin/java -Dproc_timelineserver -Xmx8192m
      

       

      Lsof shows a lot of open timeline-cache.ldb files which are referenced by ATS, even though are deleted (DEL), since they are not present when listing them .

       

      java 25519 yarn DEL REG 253,28 9438452 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000007.sst
      java 25519 yarn DEL REG 253,28 9438438 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000007.sst
      java 25519 yarn DEL REG 253,28 9438437 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000005.sst
      

       

      Looks like LevelDBCacheTimelineStore is not closing these files as the LevelDB DBIterator is not closed.

        Attachments

        1. YARN-9040.001.patch
          12 kB
          Tarun Parimi
        2. YARN-9040.002.patch
          12 kB
          Tarun Parimi

          Activity

            People

            • Assignee:
              tarunparimi Tarun Parimi
              Reporter:
              tarunparimi Tarun Parimi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: