[SPARK-1676] HDFS FileSystems continually pile up in the FS cache

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.1, 1.0.0
• Fix Version/s: 0.9.2, 1.0.0
• Component/s: Spark Core
• Assignee: Thomas Graves
• Reporter: Aaron Davidson
• Created: 30/Apr/14 04:08
• Updated: 30/Oct/17 17:09
• Resolved: 05/May/14 14:52

Description

Due to HDFS-3545, FileSystem.get() always produces (and caches) a new FileSystem when provided with a new UserGroupInformation (UGI), even if the UGI represents the same user as another UGI. This causes a buildup of FileSystem objects at an alarming rate, often one per task for something like sc.textFile(). The bug is especially hard-hitting for NativeS3FileSystem, which also maintains an open connection to S3, clogging up the system file handles.

The bug was introduced in https://github.com/apache/spark/pull/29, where doAs was made the default behavior.

A fix is not forthcoming for the general case, as UGIs do not cache well, but this problem can lead to spark clusters entering into a failed state and requiring executors be restarted.

Issue Links

• is related to: SPARK-22374 STS ran into OOM in a secure cluster