[SPARK-4838] StackOverflowError when serialization task

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.1.0
• Component/s: Spark Core, SQL
• Assignee: Unassigned
• Reporter: shenh062326
• Created: 13/Dec/14 06:08
• Updated: 08/Mar/19 07:27
• Resolved: 15/Sep/15 21:26

Description

When run a sql with more than 2000 partitions, each partition a HadoopRDD, it will cause java.lang.StackOverflowError at serialize task.
Error message from spark is:Job aborted due to stage failure: Task serialization failed: java.lang.StackOverflowError
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
......

Issue Links

• is related to: SPARK-27100 Use `Array` instead of `Seq` in `FilePartition` to prevent StackOverflowError