[BEAM-11066] DataflowRunner crashes during graph rewrites of Java wordcount example with beam_fn_api experiment

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Component/s: runner-dataflow
• Assignee: Yichi Zhang
• Reporter: Yichi Zhang
• Created: 14/Oct/20 19:03
• Updated: 30/Jan/21 05:21
• Resolved: 30/Jan/21 05:20

Description

Looks like the TextIO.write() is not properly writing windowed output in streaming mode. 

 

{{Exception in thread "main" java.lang.IllegalStateException: Found nodes that matched overrides. Matches: {Node

{fullName=WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values}

=[PTransformOverride{matcher=EqualClassPTransformMatcher

{class=class org.apache.beam.sdk.transforms.Create$Values}

, overrideFactory=org.apache.beam.runners.dataflow.DataflowRunner$StreamingFnApiCreateOverrideFactory@62923ee6}]}
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:588)
at org.apache.beam.sdk.Pipeline$1.leaveCompositeTransform(Pipeline.java:237)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:661)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:463)
at org.apache.beam.sdk.Pipeline.checkNoMoreMatches(Pipeline.java:218)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:214)
at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:1180)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:871)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:192)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185)
at org.apache.beam.examples.WordCount.main(WordCount.java:192)}}

Issue Links

• links to: GitHub Pull Request #13114