[BEAM-1177] Input DStream "bundles" should be in serialized form and include relevant metadata.

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: runner-spark
• Assignee: Amit Sela
• Reporter: Amit Sela
• Created: 18/Dec/16 13:35
• Updated: 16/May/20 13:38
• Resolved: 27/Jan/17 05:03

Description

Currently, the input partitions hold "bundles" of read elements within the mapWithStateDStream used for the read.
Since this is automatically shuffled, user-data (the read elements) should be serialized using coders to avoid breaking (if user-data is not Kryo serializable).
Even after BEAM-848 would complete, the resulting MapWithStateDStream would be checkpointed periodically and so it would still have to remain in serialized form.

Issue Links

• is related to: BEAM-979 ConcurrentModificationException exception after hours of running
• relates to: BEAM-848 Shuffle input read-values to get maximum parallelism.
• links to: GitHub Pull Request #1654
• links to: GitHub Pull Request #1654