Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19522

--executor-memory flag doesn't work in local-cluster mode

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: Deploy
    • Labels:

      Description

      bin/spark-shell --master local-cluster[2,1,2048]
      

      doesn't do what you think it does. You'll end up getting executors with 1GB each because that's the default. This is because the executor memory flag isn't actually read in local mode.

        Attachments

          Activity

            People

            • Assignee:
              andrewor14 Andrew Or
              Reporter:
              andrewor14 Andrew Or

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment