Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8335 Add streaming support to Interactive Beam
  3. BEAM-9687

Names of temporary files created by interactive runner include characters invalid on some platforms.

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • P3
    • Resolution: Fixed
    • None
    • 2.21.0
    • sdk-py-core
    • None

    Description

      Temporary files created by interactive runner in streaming scenarios include pipe '|' characters, which are not allowed in filenames on Windows platform. This causes test failures on a Windows platform:

      ====================================
      python setup.py nosetests --tests apache_beam/runners/interactive/pipeline_instrument_test.py:PipelineInstrumentTest.test_instrument_example_unbounded_pipeline_to_multiple_read_cache

      ======================================================================
      ERROR: Tests that the instrumenter works for multiple unbounded sources.
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py", line 698, in test_instrument_example_unbounded_pipeline_to_multiple_read_cache

      self._mock_write_cache([b''], cache_key)
      File "C:\projects\apache_beam\runners\interactive\pipeline_instrument_test.py", line 227, in _mock_write_cache
      ie.current_env().cache_manager().write(values, *labels)
      File "C:\projects\apache_beam\runners\interactive\caching\streaming_cache.py", line 323, in write
      with open(filepath, 'ab') as f:
      IOError: [Errno 22] invalid mode ('ab') or filename: 'c:\\users\\deft-t~1\\appdata\\local\\temp\\2\\interactive-temp-xwg5qi\\full
      pcoll_1|149781752|149781920|1
      49231600'
      ====================================

      [1]
      https://github.com/apache/beam/blob/e6b37c44d542969b6104fc97ee6f25b6f7d2ddba/sdks/python/apache_beam/runners/interactive/caching/streaming_cache.py#L323

      Attachments

        Issue Links

          Activity

            People

              rohdesam Sam Rohde
              tvalentyn Valentyn Tymofieiev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m