Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
1.0.0
-
None
-
None
Description
Recently pyspark was ported to yarn (pr 30), but when I went to try it I couldn't get it work. I was building on a redhat 6 box. I figured out that if the assembly jar file contained over 65536 files/directories then it wouldn't work. If I unjarred the assembly and removed some stuff to get it under 65536 and jarred it back up, then it would work.
It appears to only be an issue when building on a redhat box as I can build on my mac and it works just fine there.
Attachments
Issue Links
- duplicates
-
SPARK-1520 Assembly Jar with more than 65536 files won't work when compiled on JDK7 and run on JDK6
- Resolved