Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11066

DataflowRunner crashes during graph rewrites of Java wordcount example with beam_fn_api experiment

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: runner-dataflow
    • Labels:
      None

      Description

      Looks like the TextIO.write() is not properly writing windowed output in streaming mode. 

       

      {{Exception in thread "main" java.lang.IllegalStateException: Found nodes that matched overrides. Matches: {Node

      {fullName=WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values}

      =[PTransformOverride{matcher=EqualClassPTransformMatcher

      {class=class org.apache.beam.sdk.transforms.Create$Values}

      , overrideFactory=org.apache.beam.runners.dataflow.DataflowRunner$StreamingFnApiCreateOverrideFactory@62923ee6}]}
      at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkState(Preconditions.java:588)
      at org.apache.beam.sdk.Pipeline$1.leaveCompositeTransform(Pipeline.java:237)
      at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:661)
      at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
      at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
      at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:463)
      at org.apache.beam.sdk.Pipeline.checkNoMoreMatches(Pipeline.java:218)
      at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:214)
      at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:1180)
      at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:871)
      at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:192)
      at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
      at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
      at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185)
      at org.apache.beam.examples.WordCount.main(WordCount.java:192)}}

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yichi Yichi Zhang
                Reporter:
                yichi Yichi Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h