[BEAM-5775] Make the spark runner not serialize data unless spark is spilling to disk

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: Mike Kaplinskiy
• Created: 17/Oct/18 00:29
• Updated: 03/Jun/22 22:59

Description

Currently for storage level MEMORY_ONLY, Beam does not coder-ify the data. This lets Spark keep the data in memory avoiding the serialization round trip. Unfortunately the logic is fairly coarse - as soon as you switch to MEMORY_AND_DISK, Beam coder-ifys the data even though Spark might have chosen to keep the data in memory, incurring the serialization overhead.

 

Ideally Beam would serialize the data lazily - as Spark chooses to spill to disk. This would be a change in behavior when using beam, but luckily Spark has a solution for folks that want data serialized in memory - MEMORY_AND_DISK_SER will keep the data serialized.

Issue Links

• links to: GitHub Pull Request #6714
• links to: GitHub Pull Request #8371