[BEAM-9945] Use consistent element count for progress counter.

• Type: Improvement
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.21.0
• Component/s: beam-model, sdk-go, sdk-java-harness, sdk-py-harness
• Labels: Done
• Assignee: Robert Bradshaw
• Reporter: Robert Bradshaw
• Created: 09/May/20 08:17
• Updated: 06/Oct/20 21:50
• Resolved: 13/May/20 20:47

Description

This influences how the SDK communicates progress and how splitting is performed.

We currently inspect the element count of the output (which has inconsistent definitions across SDKs, see BEAM-9934). Instead, we can move to using an explicit, separate metric.

This can currently lead to incorrect progress reporting and in some cases even a crash in the UW depending on the SDK, and makes it more difficult (but not impossible) to fix element counts in the future.

Issue Links

• links to: GitHub Pull Request #11652
• links to: GitHub Pull Request #11697