[SPARK-5077] Map output statuses can still exceed spark.akka.frameSize

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.2.0, 1.3.0, 1.4.1
• Component/s: Shuffle, Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Josh Rosen
• Created: 04/Jan/15 21:55
• Updated: 17/May/20 18:31
• Resolved: 21/May/19 05:37

Description

Since HighlyCompressedMapOutputStatuses uses a bitmap for tracking empty blocks, its size is not bounded and thus Spark is still susceptible to "MapOutputTrackerMasterActor: Map output statuses
were 11141547 bytes which exceeds spark.akka.frameSize"-type errors, even in 1.2.0.

We needed to use a bitmap for tracking zero-sized blocks (see SPARK-3740; this isn't just a performance issue; it's necessary for correctness). This will require a bit more effort to fix, since we'll either have to find a way to use a fixed size / capped size encoding for MapOutputStatuses (which might require changes to let us fetch empty blocks safely) or figure out some other strategy for shipping these statues.