[BEAM-9687] Names of temporary files created by interactive runner include characters invalid on some platforms.

• Type: Sub-task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.21.0
• Component/s: sdk-py-core
• Assignee: Sam Rohde
• Reporter: Valentyn Tymofieiev
• Created: 03/Apr/20 03:13
• Updated: 12/Apr/21 21:44
• Resolved: 12/Apr/21 21:44

Description

Temporary files created by interactive runner in streaming scenarios include pipe '|' characters, which are not allowed in filenames on Windows platform. This causes test failures on a Windows platform:

====================================
python setup.py nosetests --tests apache_beam/runners/interactive/pipeline_instrument_test.py:PipelineInstrumentTest.test_instrument_example_unbounded_pipeline_to_multiple_read_cache

======================================================================
ERROR: Tests that the instrumenter works for multiple unbounded sources.
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py", line 698, in test_instrument_example_unbounded_pipeline_to_multiple_read_cache

self._mock_write_cache([b''], cache_key)
File "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py", line 227, in _mock_write_cache
ie.current_env().cache_manager().write(values, *labels)
File "C:\projects\apache_beam\runners\interactive\caching\streaming_cache.py", line 323, in write
with open(filepath, 'ab') as f:
IOError: [Errno 22] invalid mode ('ab') or filename: 'c:\\users\\deft-t~1\\appdata\\local\\temp\\2\\interactive-temp-xwg5qi\\full
pcoll_1|149781752|149781920|1
49231600'
====================================

[1]
https://github.com/apache/beam/blob/e6b37c44d542969b6104fc97ee6f25b6f7d2ddba/sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py#L323

Issue Links

• links to: GitHub Pull Request #11306