[SPARK-11789] Spark prints misleading error messages about losing executors

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 17/Nov/15 20:56
• Updated: 19/Nov/15 21:47
• Resolved: 19/Nov/15 21:47

Description

In certain cases, there are misleading ERROR messages in Spark's logs related to executors stopping:

15/11/17 11:48:53 ERROR YarnScheduler: Lost executor 4 on blah: Pending loss reason.

That shouldn't be printed, since the executor's exit reason is not yet known.

15/11/17 11:48:53 ERROR YarnScheduler: Actual reason for lost executor 4: Container container_1447778501188_0006_01_000005 exited from explicit termination request.

That should not be an ERROR, since the executor exited from an explicit request from the driver (due to dynamic allocation in this case).

Issue Links

• duplicates: SPARK-4134 Dynamic allocation: tone down scary executor lost messages when killing on purpose
• links to: [Github] Pull Request #9780 (vanzin)