[HOP-3126] Beam: re-add support for batching

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.99
• Fix Version/s: 0.99
• Component/s: API, Beam, Transforms
• Assignee: Unassigned
• Reporter: Matt Casters
• Created: 23/Jul/21 12:59
• Updated: 27/Jul/21 13:41
• Resolved: 26/Jul/21 09:52

Description

Support for batching was somehow removed from the Beam runners.

The idea is that you can set the copies of a transform to BATCH so that records gets batched up following the pipeline run configuration options:

• Streaming Hop transforms flush interval
• Hop Streaming transforms buffer size