[BEAM-5519] Spark Streaming Duplicated Encoding/Decoding Effort

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.16.0
• Component/s: runner-spark
• Labels: spark spark-streaming
• Assignee: Kyle Winkelman
• Reporter: Kyle Winkelman
• Created: 27/Sep/18 20:37
• Updated: 16/May/20 14:06
• Resolved: 19/Aug/19 15:01

Description

When using the SparkRunner in streaming mode. There is a call to groupByKey followed by a call to updateStateByKey. BEAM-1815 fixed an issue where this used to cause 2 shuffles but it still causes 2 encode/decode cycles.

Issue Links

• links to: GitHub Pull Request #6511