Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1718

pyspark doesn't work with assembly jar containing over 65536 files/dirs built on redhat

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.0.0
    • None
    • PySpark
    • None

    Description

      Recently pyspark was ported to yarn (pr 30), but when I went to try it I couldn't get it work. I was building on a redhat 6 box. I figured out that if the assembly jar file contained over 65536 files/directories then it wouldn't work. If I unjarred the assembly and removed some stuff to get it under 65536 and jarred it back up, then it would work.

      It appears to only be an issue when building on a redhat box as I can build on my mac and it works just fine there.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: