[SPARK-25766] AMCredentialRenewer can leak FS clients

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 18/Oct/18 10:20
• Updated: 17/May/20 18:14
• Resolved: 13/Feb/19 21:52

Description

AMCredentialRenewer's scheduled writeNewCredentialsToHDFS operation creates a new FS connector each time, so as to access the store with refreshed credentials

but it doesn't close it after, so any resources used by the client are kept around. This is more expensive with the cloud store connectors which create thread pools.

It should call remoteFs.close() at the end of its work.