Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8989

Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)

    XMLWordPrintableJSON

    Details

      Description

      PR/9275 changed ParDo.getSideInputs from List<PCollectionView> to Map<String, PCollectionView> which is backwards incompatible change and was released as part of Beam 2.16.0 erroneously.

      Running the Apache Nemo Quickstart fails with:

       

      Exception in thread "main" java.lang.RuntimeException: Translator private static void org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput) have failed to translate org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27Exception in thread "main" java.lang.RuntimeException: Translator private static void org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput) have failed to translate org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27 at org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:113) at org.apache.nemo.compiler.frontend.beam.PipelineVisitor.visitPrimitiveTransform(PipelineVisitor.java:46) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317) at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251) at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) at org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:80) at org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:31) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at org.apache.beam.examples.WordCount.main(WordCount.java:192)Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:109) ... 14 moreCaused by: java.lang.NoSuchMethodError: org.apache.beam.sdk.transforms.ParDo$MultiOutput.getSideInputs()Ljava/util/List; at org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(PipelineTranslator.java:236) ... 19 more

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lcwik Luke Cwik
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: