[SPARK-41599] Memory leak in FileSystem.CACHE when submitting apps to secure cluster using InProcessLauncher

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.5.0
• Component/s: Deploy, YARN
• Assignee: Xieming Li
• Reporter: Maciej Smolenski
• Created: 20/Dec/22 10:45
• Updated: 30/Jun/23 13:28
• Resolved: 30/Jun/23 13:24

Description

When submitting spark application in kerberos environment the credentials of 'current user' (UserGroupInformation.getCurrentUser()) are being modified.
Filesystem.CACHE entries contain 'current user' (with user credentials) as a key.
Submitting many spark applications using InProcessLauncher cause that FileSystem.CACHE becomes bigger and bigger.
Finally process exits because of OutOfMemory error.

Code for reproduction attached.

 

Output from running 'jmap -histo' on reproduction jvm shows that the number of FileSystem$Cache$Key increases in time:

time: #instances class
1671533274: 2 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533335: 11 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533395: 21 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533455: 30 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533515: 39 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533576: 48 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533636: 57 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533696: 66 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533757: 75 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533817: 84 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533877: 93 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533937: 102 org.apache.hadoop.fs.FileSystem$Cache$Key
1671533998: 111 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534058: 120 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534118: 135 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534178: 140 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534239: 150 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534299: 159 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534359: 168 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534419: 177 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534480: 186 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534540: 195 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534600: 204 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534661: 213 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534721: 222 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534781: 231 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534841: 240 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534902: 249 org.apache.hadoop.fs.FileSystem$Cache$Key
1671534962: 257 org.apache.hadoop.fs.FileSystem$Cache$Key
1671535022: 264 org.apache.hadoop.fs.FileSystem$Cache$Key
1671535083: 273 org.apache.hadoop.fs.FileSystem$Cache$Key
1671535143: 282 org.apache.hadoop.fs.FileSystem$Cache$Key
1671535203: 291 org.apache.hadoop.fs.FileSystem$Cache$Key

Issue Links

• links to: [Github] Pull Request #41692 (risyomei)