Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.0
-
None
Description
It looks like SPARK-19326 added a bug in the execuctor allocation maanger where it adds an extra executor when it shouldn't when we have pending speculative tasks but the target number didn't change. https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L377
It doesn't look like this is necessary since it already added in the pendingSpeculative tasks.
See the questioning of this on the PR at:
https://github.com/apache/spark/pull/18492/files#diff-b096353602813e47074ace09a3890d56R379
Attachments
Issue Links
- Blocked
-
SPARK-30511 Spark marks intentionally killed speculative tasks as pending leads to holding idle executors
-
- Resolved
-
- is caused by
-
SPARK-19326 Speculated task attempts do not get launched in few scenarios
-
- Resolved
-