[BEAM-2984] Job submission too large with embedded Beam protos

• Type: Bug
• Status: Resolved
• Priority: P0
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Component/s: runner-dataflow
• Assignee: Kenneth Knowles
• Reporter: Kenneth Knowles
• Created: 22/Sep/17 18:39
• Updated: 16/May/20 14:20
• Resolved: 22/Sep/17 19:49

Description

Empirically, naively putting context around the DoFnInfo could cause a blowup of 40%, which is too much and might cause jobs that were will under API size limits to start to fail.

There's a certain amount of wiggle room since it is hard to control the submission size anyhow, but 40% is way too much.

Issue Links

• links to: GitHub Pull Request #3887