[SPARK-32210] Failed to serialize large MapStatuses

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.4, 2.4.8, 3.0.3, 3.1.2, 3.3.0
• Fix Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Kazuyuki Tanimura
• Reporter: Yuming Wang
• Created: 07/Jul/20 13:45
• Updated: 16/Aug/21 16:13
• Resolved: 16/Aug/21 16:13

Description

Driver side exception:

20/07/07 02:22:26,366 ERROR [map-output-dispatcher-3] spark.MapOutputTrackerMaster:91 :
java.lang.NegativeArraySizeException
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:984)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply$mcV$sp(MapOutputTracker.scala:228)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:222)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:493)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
20/07/07 02:22:26,366 ERROR [map-output-dispatcher-5] spark.MapOutputTrackerMaster:91 :
java.lang.NegativeArraySizeException
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
        at org.apache.spark.MapOutputTracker$.serializeMapStatuses(MapOutputTracker.scala:984)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply$mcV$sp(MapOutputTracker.scala:228)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus$$anonfun$serializedMapStatus$2.apply(MapOutputTracker.scala:222)
        at org.apache.spark.ShuffleStatus.withWriteLock(MapOutputTracker.scala:72)
        at org.apache.spark.ShuffleStatus.serializedMapStatus(MapOutputTracker.scala:222)
        at org.apache.spark.MapOutputTrackerMaster$MessageLoop.run(MapOutputTracker.scala:493)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
20/07/07 02:22:26,366 ERROR [map-output-dispatcher-2] spark.MapOutputTrackerMaster:91 :

Issue Links

• links to: [Github] Pull Request #33721 (kazuyukitanimura)