[SPARK-13216] Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.5.0
• Component/s: DStreams, Spark Submit
• Labels: Streaming bulk-closed
• Assignee: Unassigned
• Reporter: Neelesh Srinivas Salian
• Created: 05/Feb/16 21:08
• Updated: 21/May/19 04:37
• Resolved: 21/May/19 04:37

Description

Scenario to help understand:
1) The Spark streaming job with 12 executors was initiated with checkpointing enabled.
2) In version 1.3, the user was able to append the number of executors to 20 using --num-executors but was unable to do so in version 1.5.

In 1.5, the spark application still runs with 13 executors (1 for driver and 12 executors).
There is a need to start from the checkpoint itself and not restart the application to avoid the loss of information.

3) Checked the code in 1.3 and 1.5, which shows the command ''--num-executors" has been deprecated.

Any thoughts on this? Not sure if anyone hit this one specifically before.