Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31142

Remove useless conf set in pyspark context

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: PySpark, Spark Core
    • Labels:
      None

      Description

      In Pyspark/context.py, we extract the configuration of the "spark.executorEnv" prefix from conf and put it in env. But in PythonWorkerFactory, we have already obtained these environment variables from ProgressBuilder, thus we do not need the extra copy.
      In addition, in some special environment deployments, user-configured environment variables such as "spark.executorEnv.Path=/usr/lib:$PATH" will be automatically parsed as actual machine confs. If we still directly copy and overwrite it, the configuration will be a wrong config.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jackey Lee Jackey Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: