Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3634

Python modules added through addPyFile should take precedence over system modules

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.2, 1.1.0
    • 1.2.0
    • PySpark
    • None

    Description

      Python modules added through SparkContext.addPyFile() are currently appended to sys.path; this is probably the opposite of the behavior that we want, since it causes system versions of modules to take precedence over versions explicitly added by users.

      To fix this, we should change the sys.path manipulation code in context.py and worker.py to prepend files to sys.path.

      Attachments

        Activity

          People

            Unassigned Unassigned
            joshrosen Josh Rosen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: