[SPARK-4134] Dynamic allocation: tone down scary executor lost messages when killing on purpose

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.6.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Andrew Or
• Created: 29/Oct/14 16:08
• Updated: 23/Nov/15 17:34
• Resolved: 23/Nov/15 17:34

Description

After SPARK-3822 goes in, we are now able to dynamically kill executors after an application has started. However, when we do that we get a ton of scary error messages telling us that we've done wrong somehow. It would be good to detect when this is the case and prevent these messages from surfacing.

This maybe difficult, however, because the connection manager tends to be quite verbose in unconditionally logging disconnection messages. This is a very nice-to-have for 1.2 but certainly not a blocker.

Issue Links

• is duplicated by: SPARK-11789 Spark prints misleading error messages about losing executors
• links to: [Github] Pull Request #6310 (andrewor14)
• links to: [Github] Pull Request #9780 (vanzin)