Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13216

Spark streaming application not honoring --num-executors in restarting of an application from a checkpoint

    XMLWordPrintableJSON

    Details

      Description

      Scenario to help understand:
      1) The Spark streaming job with 12 executors was initiated with checkpointing enabled.
      2) In version 1.3, the user was able to append the number of executors to 20 using --num-executors but was unable to do so in version 1.5.

      In 1.5, the spark application still runs with 13 executors (1 for driver and 12 executors).
      There is a need to start from the checkpoint itself and not restart the application to avoid the loss of information.

      3) Checked the code in 1.3 and 1.5, which shows the command ''--num-executors" has been deprecated.

      Any thoughts on this? Not sure if anyone hit this one specifically before.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              neelesh77 Neelesh Srinivas Salian
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: