Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12416

Python Kafka transforms are failing due to "No Runner was specified"

Details

    • Bug
    • Status: Resolved
    • P1
    • Resolution: Fixed
    • None
    • 2.30.0
    • cross-language
    • None

    Description

      Transforms are failing during x-lang expansion.

       

       

      {{RuntimeError: java.lang.IllegalArgumentException: No Runner was specified and the DirectRunner was not found on the classpath.
      Specify a runner by either:
      Explicitly specifying a runner by providing the 'runner' property
      Adding the DirectRunner to the classpath
      Calling 'PipelineOptions.setRunner(PipelineRunner)' directly
      at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:302)
      at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:288)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:605)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:546)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:171)
      at com.sun.proxy.$Proxy4.getRunner(Unknown Source)
      at org.apache.beam.sdk.io.kafka.KafkaIO$Read.runnerRequiresLegacyRead(KafkaIO.java:1266)
      at org.apache.beam.sdk.io.kafka.KafkaIO$Read.expand(KafkaIO.java:1253)
      at org.apache.beam.sdk.io.kafka.KafkaIO$Read.expand(KafkaIO.java:600)
      at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
      at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:482)
      at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
      at org.apache.beam.sdk.io.kafka.KafkaIO$TypedWithoutMetadata.expand(KafkaIO.java:1528)
      at org.apache.beam.sdk.io.kafka.KafkaIO$TypedWithoutMetadata.expand(KafkaIO.java:1517)
      at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:548)
      at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:499)
      at org.apache.beam.sdk.expansion.service.ExpansionService$TransformProvider.apply(ExpansionService.java:360)
      at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:436)
      at org.apache.beam.sdk.expansion.service.ExpansionService.expand(ExpansionService.java:491)
      at org.apache.beam.model.expansion.v1.ExpansionServiceGrpc$MethodHandlers.invoke(ExpansionServiceGrpc.java:219)
      at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
      at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
      at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:797)
      at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
      at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)}}

      {{}}

      Seems to be due to https://github.com/apache/beam/commit/c097d8424678813901ac46f2fe674cfe71e67430

      {{}}

      We'll have to cherry-pick a fix to 2.30.0 since above commit was cherry-picked. Boyuan is looking into it.

      Attachments

        Activity

          People

            boyuanz Boyuan Zhang
            chamikara Chamikara Madhusanka Jayalath
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2h
                2h