[SPARK-21321] Spark very verbose on shutdown confusing users

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Spark Core
• Assignee: Jong Yoon Lee
• Reporter: Jong Yoon Lee
• Created: 05/Jul/17 16:02
• Updated: 17/Jul/17 18:14
• Resolved: 17/Jul/17 18:14

Description

On shutdown, spark can be very verbose and can spit out errors that causes the user to be confused.

If possible, we should not print those out and just ignore them by changing the changing the log level from WARNING to DEBUG.

Also shutdown of spark can take a long time because of the backlog of events in the event queue ( warnings like: Message $message dropped. ${e.getMessage}).

This happens more with dynamic allocation on.

I am suggesting to
1. change the log level when the shutdown is happening and the RPC connections are closed(RpcEnvStoppedException).

2. Clear the event queue when RPC module is stopped and spark is shutting down.

Issue Links

• links to: [Github] Pull Request #18547 (yoonlee95)