Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1753

PySpark on YARN does not work on assembly jar built on Red Hat based OS

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.0
    • 1.0.0
    • PySpark, Spark Core
    • None

    Description

      If the jar is built on a Red Hat based OS, the additional python files included in the jar cannot be accessed. This means PySpark doesn't work on YARN because in this mode it relies on the python files within this jar.

      I have confirmed that my Java, Scala, and maven versions are all exactly the same on my CentOS environment and on my local OSX environment, and the former does not work. Thomas Graves also struggled with the same problem.

      Until a fix is found, we should at the very least document this peculiarity.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            andrewor14 Andrew Or
            andrewor14 Andrew Or
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment