Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-11433

Dataflow Multi-language pipelines broken for Java 11 users

    XMLWordPrintableJSON

    Details

      Description

      I get [1] for Dataflow multi-language Kafka example [2] when running the pipeline with Java11 installed in the computer and with Java11 containers.

      Due to recent changes related to JDK specific container naming, Beam Docker environments default to using Java version specific containers. This means that Dataflow multi-language pipelines will end up using Java 11 containers for users that use Java 11. Seems like these containers are currently broken for Dataflow.

       

      See [2] for instructions for reproducing (run with Java11 containers instead of Java8 containers)

      Emily Ye, what would be a good solution for Dataflow till we support Java 11 ? I suspect we'll run into similar failures if we just run regular Java pipelines on UW with Java 11 containers.

      This should be a blocker for Beam 2.27.0 since this breaks Dataflow multi-language pipelines for Java 11 users.

      cc: Pablo Estrada Robert Bradshaw Kenneth Knowles

      [1]

      Error message from worker: generic::unknown: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74) org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:706) org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:360) org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:346) org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:178) org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:152) org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:239) org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200) org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200) org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:200) org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:505) org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:279) org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:567) org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:274) org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:173) org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:157) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.io.InvalidClassException: org.apache.beam.sdk.io.Read$UnboundedSourceAsSDFWrapperFn; local class incompatible: stream classdesc serialVersionUID = -1752193696517031629, local class serialVersionUID = 3454873399160940695 java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689) java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2012) java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862) java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169) java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679) java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2464) java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2358) java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2196) java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1679) 

      [2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/kafkataxi/README.md

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                emilymye Emily Ye
                Reporter:
                chamikara Chamikara Madhusanka Jayalath
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h