[FLINK-33595] StreamDependencyTests.test_set_requirements_with_cached_directory failed with ModuleNotFoundError

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.18.0
• Component/s: API / Python
• Labels: test-stability
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 20/Nov/23 07:30
• Updated: 25/Mar/24 14:24

Description

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=54682&view=logs&j=b53e1644-5cb4-5a3b-5d48-f523f39bcf06&t=b68c9f5c-04c9-5c75-3862-a3a27aabbce3&l=24663

[...]
Nov 18 02:53:13 E                   ModuleNotFoundError: No module named 'python_package1'
Nov 18 02:53:13 E                   
Nov 18 02:53:13 E                   	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
Nov 18 02:53:13 E                   	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
Nov 18 02:53:13 E                   	at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:61)
Nov 18 02:53:13 E                   	at org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor$ActiveBundle.close(SdkHarnessClient.java:504)
Nov 18 02:53:13 E                   	at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory$1.close(DefaultJobBundleFactory.java:555)
Nov 18 02:53:13 E                   	at org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.finishBundle(BeamPythonFunctionRunner.java:421)
Nov 18 02:53:13 E                   	... 7 more
Nov 18 02:53:13 E                   Caused by: java.lang.RuntimeException: Error received from SDK harness for instruction 1: Traceback (most recent call last):
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 295, in _execute
Nov 18 02:53:13 E                       response = task()
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 370, in <lambda>
Nov 18 02:53:13 E                       lambda: self.create_worker().do_instruction(request), request)
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 630, in do_instruction
Nov 18 02:53:13 E                       getattr(request, request_type), request.instruction_id)
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 667, in process_bundle
Nov 18 02:53:13 E                       bundle_processor.process_bundle(instruction_id))
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1062, in process_bundle
Nov 18 02:53:13 E                       element.data)
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/.tox/py37-cython/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 231, in process_encoded
Nov 18 02:53:13 E                       self.output(decoded_value)
Nov 18 02:53:13 E                     File "apache_beam/runners/worker/operations.py", line 526, in apache_beam.runners.worker.operations.Operation.output
Nov 18 02:53:13 E                     File "apache_beam/runners/worker/operations.py", line 528, in apache_beam.runners.worker.operations.Operation.output
Nov 18 02:53:13 E                     File "apache_beam/runners/worker/operations.py", line 237, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
Nov 18 02:53:13 E                     File "apache_beam/runners/worker/operations.py", line 240, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
Nov 18 02:53:13 E                     File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 169, in pyflink.fn_execution.beam.beam_operations_fast.FunctionOperation.process
Nov 18 02:53:13 E                       with self.scoped_process_state:
Nov 18 02:53:13 E                     File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 196, in pyflink.fn_execution.beam.beam_operations_fast.FunctionOperation.process
Nov 18 02:53:13 E                       self.process_element(input_processor.next())
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/pyflink/fn_execution/table/operations.py", line 102, in process_element
Nov 18 02:53:13 E                       return self.func(value)
Nov 18 02:53:13 E                     File "<string>", line 1, in <lambda>
Nov 18 02:53:13 E                     File "/__w/2/s/flink-python/pyflink/table/tests/test_dependency.py", line 184, in add_one
Nov 18 02:53:13 E                       from python_package1 import plus
Nov 18 02:53:13 E                   ModuleNotFoundError: No module named 'python_package1'
Nov 18 02:53:13 E                   
Nov 18 02:53:13 E                   	at org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:180)
Nov 18 02:53:13 E                   	at org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:160)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:332)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:315)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
Nov 18 02:53:13 E                   	at org.apache.beam.vendor.grpc.v1p48p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
Nov 18 02:53:13 E                   	... 3 more 

Issue Links

• is related to: FLINK-26644 python StreamExecutionEnvironmentTests.test_generate_stream_graph_with_dependencies failed on azure