[SPARK-16138] YarnAllocator tries to cancel executor requests when we have none

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Peter Ableda
• Reporter: Peter Ableda
• Created: 22/Jun/16 15:48
• Updated: 17/May/20 18:13
• Resolved: 23/Jun/16 14:01

Description

Easy to reproduce:
$ spark-shell
> sc.parallelize(1 to 10000).count()

After the computation finishes we have 0 pending tasks so we will need 0 executors.

16/06/21 04:07:41 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
16/06/21 04:07:41 INFO yarn.YarnAllocator: Canceling requests for 0 executor containers
16/06/21 04:07:41 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
...
16/06/21 04:08:41 INFO yarn.YarnAllocator: Canceling requests for 0 executor containers
16/06/21 04:08:41 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
16/06/21 04:08:41 INFO yarn.ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 1.
16/06/21 04:08:48 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
16/06/21 04:08:48 INFO yarn.YarnAllocator: Canceling requests for 0 executor containers
16/06/21 04:08:48 WARN yarn.YarnAllocator: Expected to find pending requests, but found none.
16/06/21 04:08:48 INFO yarn.ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 2.

Issue Links

• links to: [Github] Pull Request #13850 (peterableda)