[SPARK-22374] STS ran into OOM in a secure cluster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 27/Oct/17 23:50
• Updated: 25/May/21 01:55
• Resolved: 25/May/21 01:38

Description

In a secure cluster, FileSystem.CACHE grows indefinitely.

ENVIRONMENT
1. `spark.yarn.principal` and `spark.yarn.keytab` is used.
2. Spark Thrift Server run with `doAs` false.

<property>
  <name>hive.server2.enable.doAs</name>
  <value>false</value>
</property>

With 6GB (-Xmx6144m) options, `HiveConf` consumes 4GB inside FileSystem.CACHE.

20,030 instances of "org.apache.hadoop.hive.conf.HiveConf", loaded by "sun.misc.Launcher$AppClassLoader @ 0x64001c160" occupy 4,418,101,352 (73.42%) bytes. These instances are referenced from one instance of "java.util.HashMap$Node[]", loaded by "<system class loader>"

Please see the attached images.

Issue Links

• relates to: HDFS-3545 DFSClient leak due to malfunctioning of FileSystem Cache
• relates to: SPARK-1676 HDFS FileSystems continually pile up in the FS cache
• links to: [Github] Pull Request #19593 (dongjoon-hyun)