Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10921

Interactive Runner Python unit tests are flaky on Windows

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Over the past few days python unit tests have been failing frequently. The errors always seem to occur when cleaning up the interactive environment:

      ...........                                                              [100%]
      ================================== FAILURES ===================================
      _ PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection _
      [gw2] win32 -- Python 3.5.4 d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
      
      self = <apache_beam.runners.interactive.pipeline_instrument_test.PipelineInstrumentTest testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>
      
          def setUp(self):
      >     ie.new_env()
      
      apache_beam\runners\interactive\pipeline_instrument_test.py:46: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      apache_beam\runners\interactive\interactive_environment.py:117: in new_env
          _interactive_beam_env.cleanup()
      apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
          cache_manager.cleanup()
      apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
          shutil.rmtree(self._cache_dir)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
          return _rmtree_unsafe(path, onerror)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in _rmtree_unsafe
          _rmtree_unsafe(fullname, onerror)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in _rmtree_unsafe
          onerror(os.unlink, fullname, sys.exc_info())
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      path = 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
      onerror = <function rmtree.<locals>.onerror at 0x000001D6C3E5C7B8>
      
          def _rmtree_unsafe(path, onerror):
              try:
                  if os.path.islink(path):
                      # symlinks to directories are forbidden, see bug #1669
                      raise OSError("Cannot call rmtree on a symbolic link")
              except OSError:
                  onerror(os.path.islink, path, sys.exc_info())
                  # can't continue even if onerror hook returns
                  return
              names = []
              try:
                  names = os.listdir(path)
              except OSError:
                  onerror(os.listdir, path, sys.exc_info())
              for name in names:
                  fullname = os.path.join(path, name)
                  try:
                      mode = os.lstat(fullname).st_mode
                  except OSError:
                      mode = 0
                  if stat.S_ISDIR(mode):
                      _rmtree_unsafe(fullname, onerror)
                  else:
                      try:
      >                   os.unlink(fullname)
      E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
      
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError
      

        Attachments

          Activity

            People

            • Assignee:
              rohdesam Sam Rohde
              Reporter:
              bhulette Brian Hulette

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 8h
                8h

                  Issue deployment