[SPARK-28676] Avoid Excessive logging from ContextCleaner

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.3, 2.4.3, 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Ajith S
• Reporter: Ajith S
• Created: 09/Aug/19 08:53
• Updated: 09/Aug/19 22:54
• Resolved: 09/Aug/19 22:49

Description

In high workload environments, ContextCleaner seems to have excessive logging at INFO level which do not give much information. In one Particular case we see that ``INFO ContextCleaner: Cleaned accumulator`` message is 25-30% of the generated logs. We can log this information for cleanup in DEBUG level instead.

Issue Links

• links to: GitHub Pull Request #25396