Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3634

Python modules added through addPyFile should take precedence over system modules

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.2, 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      Python modules added through SparkContext.addPyFile() are currently appended to sys.path; this is probably the opposite of the behavior that we want, since it causes system versions of modules to take precedence over versions explicitly added by users.

      To fix this, we should change the sys.path manipulation code in context.py and worker.py to prepend files to sys.path.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: