[BEAM-12222] Dataflow side input translation "Unknown producer for value"

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.30.0
• Component/s: runner-dataflow
• Assignee: Kenneth Knowles
• Reporter: Kenneth Knowles
• Created: 25/Apr/21 15:04
• Updated: 13/May/21 19:04
• Resolved: 13/May/21 18:54

Description

I have identified a seemingly nondeterministic issue in Dataflow translation, where pipelines with side inputs sometimes are translated in the wrong order.

java.lang.NullPointerException: Unknown producer for value SimplePCollectionView{tag=Tag<org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.<init>:1221#4dca087078898728>} while translating step TfIdf.ComputeTfIdf/Combine.globally(Count)/ProduceDefault
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1227)

Seen on https://ci-beam.apache.org/job/beam_PostCommit_Java_Examples_Dataflow_V2_PR/32/testReport/junit/org.apache.beam.examples.complete/TfIdfIT/testE2ETfIdf/ and also other changes. I think the change itself is just triggering the nondeterministic problem.

So there is a lurking problem with side inputs overall.

Issue Links

• links to: GitHub Pull Request #14641
• links to: GitHub Pull Request #14687