[BEAM-10627] tests fails on windows - interactive tests fails due to FileNotFoundError

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Won't Fix
• Fix Version/s: Missing
• Component/s: sdk-py-core, testing
• Assignee: Unassigned
• Reporter: Tobiasz Kedzierski
• Created: 03/Aug/20 11:51
• Updated: 13/Jan/22 21:49
• Resolved: 14/Sep/21 23:24

Description

Failing tests:

apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_basic
apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_wordcount
apache_beam.runners.interactive.interactive_beam_test.InteractiveBeamTest.test_show_always_watch_given_pcolls
apache_beam.runners.interactive.interactive_beam_test.InteractiveBeamTest.test_show_mark_pcolls_computed_when_done

Link to the github workflow run with mentioned error:
https://github.com/TobKed/beam/runs/937336438?check_suite_focus=true

partial log:
2020-08-02T11:05:43.5852779Z ___________ InteractiveBeamTest.test_show_always_watch_given_pcolls ___________
2020-08-02T11:05:43.5853476Z [gw3] win32 – Python 3.5.4 d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
2020-08-02T11:05:43.5853847Z
2020-08-02T11:05:43.5855313Z self = <apache_beam.runners.interactive.interactive_beam_test.InteractiveBeamTest testMethod=test_show_always_watch_given_pcolls>
2020-08-02T11:05:43.5855658Z
2020-08-02T11:05:43.5855975Z def test_show_always_watch_given_pcolls(self):
2020-08-02T11:05:43.5856278Z p = beam.Pipeline(ir.InteractiveRunner())
2020-08-02T11:05:43.5856566Z # pylint: disable=range-builtin-not-iterating
2020-08-02T11:05:43.5856845Z pcoll = p | 'Create' >> beam.Create(range(10))
2020-08-02T11:05:43.5857355Z # The pcoll is not watched since watch(locals()) is not explicitly called.
2020-08-02T11:05:43.5858106Z self.assertFalse(pcoll in _get_watched_pcollections_with_variable_names())
2020-08-02T11:05:43.5858620Z # The call of show watches pcoll.
2020-08-02T11:05:43.5859235Z > ib.show(pcoll)
2020-08-02T11:05:43.5859475Z
2020-08-02T11:05:43.5860015Z apache_beam\runners\interactive\interactive_beam_test.py:96:
2020-08-02T11:05:43.5861024Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-08-02T11:05:43.5861944Z apache_beam\runners\interactive\utils.py:205: in run_within_progress_indicator
2020-08-02T11:05:43.5862682Z return func(*args, **kwargs)
2020-08-02T11:05:43.5863214Z apache_beam\runners\interactive\interactive_beam.py:411: in show
2020-08-02T11:05:43.5863760Z result = pf.PipelineFragment(list(pcolls), user_pipeline.options).run()
2020-08-02T11:05:43.5864291Z apache_beam\runners\interactive\pipeline_fragment.py:113: in run
2020-08-02T11:05:43.5864746Z return self.deduce_fragment().run()
2020-08-02T11:05:43.5865292Z apache_beam\pipeline.py:521: in run
2020-08-02T11:05:43.5865633Z allow_proto_holders=True).run(False)
2020-08-02T11:05:43.5866159Z apache_beam\pipeline.py:534: in run
2020-08-02T11:05:43.5866638Z return self.runner.run_pipeline(self, self._options)
2020-08-02T11:05:43.5867299Z apache_beam\runners\interactive\interactive_runner.py:194: in run_pipeline
2020-08-02T11:05:43.5867667Z pipeline_to_execute.run(), pipeline_instrument)
2020-08-02T11:05:43.5868119Z apache_beam\pipeline.py:534: in run
2020-08-02T11:05:43.5868627Z return self.runner.run_pipeline(self, self._options)
2020-08-02T11:05:43.5869401Z apache_beam\runners\direct\direct_runner.py:119: in run_pipeline
2020-08-02T11:05:43.5869735Z return runner.run_pipeline(pipeline, options)
2020-08-02T11:05:43.5870201Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:176: in run_pipeline
2020-08-02T11:05:43.5870665Z pipeline.to_runner_api(default_environment=self._default_environment))
2020-08-02T11:05:43.5871520Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:186: in run_via_runner_api
2020-08-02T11:05:43.5871987Z return self.run_stages(stage_context, stages)
2020-08-02T11:05:43.5872612Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:344: in run_stages
2020-08-02T11:05:43.5872918Z bundle_context_manager,
2020-08-02T11:05:43.5873512Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:523: in _run_stage
2020-08-02T11:05:43.5873851Z bundle_manager)
2020-08-02T11:05:43.5874423Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:561: in _run_bundle
2020-08-02T11:05:43.5875072Z data_input, data_output, input_timers, expected_timer_output)
2020-08-02T11:05:43.5875740Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:945: in process_bundle
2020-08-02T11:05:43.5876099Z timer_inputs)):
2020-08-02T11:05:43.5877043Z c:\hostedtoolcache\windows\python\3.5.4\x64\lib\concurrent\futures_base.py:556: in result_iterator
2020-08-02T11:05:43.5877296Z yield future.result()
2020-08-02T11:05:43.5878056Z c:\hostedtoolcache\windows\python\3.5.4\x64\lib\concurrent\futures_base.py:405: in result
2020-08-02T11:05:43.5878412Z return self.__get_result()
2020-08-02T11:05:43.5879053Z c:\hostedtoolcache\windows\python\3.5.4\x64\lib\concurrent\futures_base.py:357: in __get_result
2020-08-02T11:05:43.5879419Z raise self._exception
2020-08-02T11:05:43.5879879Z apache_beam\utils\thread_pool_executor.py:44: in run
2020-08-02T11:05:43.5880617Z self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
2020-08-02T11:05:43.5881245Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:941: in execute
2020-08-02T11:05:43.5881486Z dry_run)
2020-08-02T11:05:43.5882151Z apache_beam\runners\portability\fn_api_runner\fn_runner.py:841: in process_bundle
2020-08-02T11:05:43.5882648Z result_future = self._worker_handler.control_conn.push(process_bundle_req)
2020-08-02T11:05:43.5883269Z apache_beam\runners\portability\fn_api_runner\worker_handlers.py:352: in push
2020-08-02T11:05:43.5883663Z response = self.worker.do_instruction(request)
2020-08-02T11:05:43.5884260Z apache_beam\runners\worker\sdk_worker.py:483: in do_instruction
2020-08-02T11:05:43.5884688Z getattr(request, request_type), request.instruction_id)
2020-08-02T11:05:43.5885411Z apache_beam\runners\worker\sdk_worker.py:518: in process_bundle
2020-08-02T11:05:43.5885697Z bundle_processor.process_bundle(instruction_id))
2020-08-02T11:05:43.5886325Z apache_beam\runners\worker\bundle_processor.py:978: in process_bundle
2020-08-02T11:05:43.5886632Z element.data)
2020-08-02T11:05:43.5887166Z apache_beam\runners\worker\bundle_processor.py:218: in process_encoded
2020-08-02T11:05:43.5887466Z self.output(decoded_value)
2020-08-02T11:05:43.5888015Z apache_beam\runners\worker\operations.py:332: in output
2020-08-02T11:05:43.5888562Z cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
2020-08-02T11:05:43.5889136Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5889440Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5890038Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5890501Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5890938Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5891219Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5891754Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5892205Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5892742Z apache_beam\runners\common.py:570: in invoke_process
2020-08-02T11:05:43.5893286Z windowed_value, self.process_method(windowed_value.value))
2020-08-02T11:05:43.5893773Z apache_beam\runners\common.py:1371: in process_outputs
2020-08-02T11:05:43.5894086Z self.main_receivers.receive(windowed_value)
2020-08-02T11:05:43.5894677Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5895258Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5895850Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5896279Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5896853Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5897261Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5897794Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5898242Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5898818Z apache_beam\runners\common.py:570: in invoke_process
2020-08-02T11:05:43.5899253Z windowed_value, self.process_method(windowed_value.value))
2020-08-02T11:05:43.5899693Z apache_beam\runners\common.py:1371: in process_outputs
2020-08-02T11:05:43.5900009Z self.main_receivers.receive(windowed_value)
2020-08-02T11:05:43.5900593Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5900907Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5901487Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5901895Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5902425Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5902747Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5903273Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5903722Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5904244Z apache_beam\runners\common.py:570: in invoke_process
2020-08-02T11:05:43.5906019Z windowed_value, self.process_method(windowed_value.value))
2020-08-02T11:05:43.5907258Z apache_beam\runners\common.py:1371: in process_outputs
2020-08-02T11:05:43.5907654Z self.main_receivers.receive(windowed_value)
2020-08-02T11:05:43.5908042Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5908684Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5909344Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5910421Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5910866Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5911116Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5911450Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5911759Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5912095Z apache_beam\runners\common.py:743: in invoke_process
2020-08-02T11:05:43.5912432Z windowed_value, additional_args, additional_kwargs)
2020-08-02T11:05:43.5913114Z apache_beam\runners\common.py:868: in _invoke_process_per_window
2020-08-02T11:05:43.5913534Z self.threadsafe_watermark_estimator)
2020-08-02T11:05:43.5914166Z apache_beam\runners\common.py:1371: in process_outputs
2020-08-02T11:05:43.5914896Z self.main_receivers.receive(windowed_value)
2020-08-02T11:05:43.5915860Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5916467Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5917277Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5918044Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5918715Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5919012Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5919716Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5920223Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5920932Z apache_beam\runners\common.py:743: in invoke_process
2020-08-02T11:05:43.5922244Z windowed_value, additional_args, additional_kwargs)
2020-08-02T11:05:43.5922797Z apache_beam\runners\common.py:868: in _invoke_process_per_window
2020-08-02T11:05:43.5923112Z self.threadsafe_watermark_estimator)
2020-08-02T11:05:43.5923454Z apache_beam\runners\common.py:1371: in process_outputs
2020-08-02T11:05:43.5923791Z self.main_receivers.receive(windowed_value)
2020-08-02T11:05:43.5924916Z apache_beam\runners\worker\operations.py:195: in receive
2020-08-02T11:05:43.5925359Z self.consumer.process(windowed_value)
2020-08-02T11:05:43.5925741Z apache_beam\runners\worker\operations.py:671: in process
2020-08-02T11:05:43.5926284Z delayed_application = self.dofn_runner.process(o)
2020-08-02T11:05:43.5926952Z apache_beam\runners\common.py:1215: in process
2020-08-02T11:05:43.5927818Z self._reraise_augmented(exn)
2020-08-02T11:05:43.5928947Z apache_beam\runners\common.py:1294: in _reraise_augmented
2020-08-02T11:05:43.5929923Z raise_with_traceback(new_exn)
2020-08-02T11:05:43.5930261Z target\.tox\py35-win\lib\site-packages\future\utils_init_.py:446: in raise_with_traceback
2020-08-02T11:05:43.5930538Z raise exc.with_traceback(traceback)
2020-08-02T11:05:43.5930828Z apache_beam\runners\common.py:1213: in process
2020-08-02T11:05:43.5931267Z return self.do_fn_invoker.invoke_process(windowed_value)
2020-08-02T11:05:43.5931563Z apache_beam\runners\common.py:743: in invoke_process
2020-08-02T11:05:43.5932030Z windowed_value, additional_args, additional_kwargs)
2020-08-02T11:05:43.5932856Z apache_beam\runners\common.py:867: in _invoke_process_per_window
2020-08-02T11:05:43.5933758Z self.process_method(*args_for_process),
2020-08-02T11:05:43.5934275Z apache_beam\io\iobase.py:1061: in process
2020-08-02T11:05:43.5935058Z self.writer = self.sink.open_writer(init_result, str(uuid.uuid4()))
2020-08-02T11:05:43.5935807Z apache_beam\options\value_provider.py:135: in _f
2020-08-02T11:05:43.5936345Z return fnc(self, *args, **kwargs)
2020-08-02T11:05:43.5937517Z apache_beam\io\filebasedsink.py:196: in open_writer
2020-08-02T11:05:43.5938203Z return FileBasedSinkWriter(self, writer_path)
2020-08-02T11:05:43.5938575Z apache_beam\io\filebasedsink.py:417: in _init_
2020-08-02T11:05:43.5939191Z self.temp_handle = self.sink.open(temp_shard_path)
2020-08-02T11:05:43.5939486Z apache_beam\io\textio.py:401: in open
2020-08-02T11:05:43.5939961Z file_handle = super(_TextSink, self).open(temp_path)
2020-08-02T11:05:43.5940778Z apache_beam\options\value_provider.py:135: in _f
2020-08-02T11:05:43.5941063Z return fnc(self, *args, **kwargs)
2020-08-02T11:05:43.5941626Z apache_beam\io\filebasedsink.py:138: in open
2020-08-02T11:05:43.5942248Z return FileSystems.create(temp_path, self.mime_type, self.compression_type)
2020-08-02T11:05:43.5942734Z apache_beam\io\filesystems.py:224: in create
2020-08-02T11:05:43.5943258Z return filesystem.create(path, mime_type, compression_type)
2020-08-02T11:05:43.5943943Z apache_beam\io\localfilesystem.py:168: in create
2020-08-02T11:05:43.5944260Z return self._path_open(path, 'wb', mime_type, compression_type)
2020-08-02T11:05:43.5946106Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-08-02T11:05:43.5946356Z
2020-08-02T11:05:43.5946637Z self = <apache_beam.io.localfilesystem.LocalFileSystem object at 0x0000017D1F14FEF0>
2020-08-02T11:05:43.5947908Z path = 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-9as6naq51636906604248\\full\\beam-temp-47e439f812-16...8ecd4af11eabb14000d3a7b9d20
130e2734-6324-4fb8-9383-4d46b2ece316.47e439f812-1636908653816-1636908654152-1636906604248'
2020-08-02T11:05:43.5948751Z mode = 'wb', mime_type = 'text/plain', compression_type = 'uncompressed'
2020-08-02T11:05:43.5949030Z
2020-08-02T11:05:43.5949268Z def _path_open(
2020-08-02T11:05:43.5949544Z self,
2020-08-02T11:05:43.5949775Z path,
2020-08-02T11:05:43.5950052Z mode,
2020-08-02T11:05:43.5950403Z mime_type='application/octet-stream',
2020-08-02T11:05:43.5950753Z compression_type=CompressionTypes.AUTO):
2020-08-02T11:05:43.5951011Z """Helper functions to open a file in the provided mode.
2020-08-02T11:05:43.5951286Z """
2020-08-02T11:05:43.5951520Z compression_type = FileSystem._get_compression_type(path, compression_type)
2020-08-02T11:05:43.5951798Z > raw_file = io.open(path, mode)
2020-08-02T11:05:43.5953920Z E RuntimeError: FileNotFoundError: [Errno 2] No such file or directory: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-9as6naq51636906604248\\full\\beam-temp-47e439f812-1636908653816-1636908654152-1636906604248-45f788ecd4af11eabb14000d3a7b9d20
130e2734-6324-4fb8-9383-4d46b2ece316.47e439f812-1636908653816-1636908654152-1636906604248' [while running 'ref_AppliedPTransform_AppliedPTransform_1/_WriteCache_47e439f812-1636908653816-1636908654152-1636906604248/Write/Write/WriteImpl/WriteBundles']

Issue Links

• duplicates: BEAM-10514 Make sure Interactive Beam cache file path length does not exceed OS limits
• relates to: BEAM-10623 Add workflow to run Beam python tests on Linux/Windows/Mac platforms