Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25021

Add spark.executor.pyspark.memory support to Kubernetes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      SPARK-25004 adds spark.executor.pyspark.memory to control the memory allocation for PySpark and updates YARN to add this memory to its container requests. Kubernetes should do something similar to account for the python memory allocation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ifilonenko Ilan Filonenko
                Reporter:
                rdblue Ryan Blue
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: