[YARN-9040] LevelDBCacheTimelineStore in ATS 1.5 leaks native memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.0.4, 3.1.2, 3.3.0, 2.8.6, 3.2.1, 2.9.3
• Component/s: timelineserver
• Target Version/s: 3.0.4, 3.1.2, 3.3.0, 2.8.6, 3.2.1, 2.9.3
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 20/Nov/18 20:35
• Updated: 17/Dec/18 17:11
• Resolved: 17/Dec/18 17:11

Description

When LevelDBCacheTimelineStore from YARN-4219 is used as ATS 1.5 entity caching storage, we observe memory leak due to leveldb files even after the fix of YARN-5368 .

Top output shows 0.024TB (25GB) RES, even though heap size is only 8GB.

 

 

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25519 yarn 20 0 33.024g 0.024t 41468 S 6.2 26.0 21:07.39 /usr/java/default/bin/java -Dproc_timelineserver -Xmx8192m

 

Lsof shows a lot of open timeline-cache.ldb files which are referenced by ATS, even though are deleted (DEL), since they are not present when listing them .

 

java 25519 yarn DEL REG 253,28 9438452 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000007.sst
java 25519 yarn DEL REG 253,28 9438438 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000007.sst
java 25519 yarn DEL REG 253,28 9438437 /var/yarn/timeline/timelineEntityGroupId_1542280269959_55569_dag_1542280269959_55569_2-timeline-cache.ldb/000005.sst

 

Looks like LevelDBCacheTimelineStore is not closing these files as the LevelDB DBIterator is not closed.