Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10921

Python unit tests on windows flaky

    XMLWordPrintableJSON

    Details

      Description

      Over the past few days python unit tests have been failing frequently. The errors always seem to occur when cleaning up the interactive environment:

      ...........                                                              [100%]
      ================================== FAILURES ===================================
      _ PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection _
      [gw2] win32 -- Python 3.5.4 d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe
      
      self = <apache_beam.runners.interactive.pipeline_instrument_test.PipelineInstrumentTest testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>
      
          def setUp(self):
      >     ie.new_env()
      
      apache_beam\runners\interactive\pipeline_instrument_test.py:46: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      apache_beam\runners\interactive\interactive_environment.py:117: in new_env
          _interactive_beam_env.cleanup()
      apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
          cache_manager.cleanup()
      apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
          shutil.rmtree(self._cache_dir)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
          return _rmtree_unsafe(path, onerror)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in _rmtree_unsafe
          _rmtree_unsafe(fullname, onerror)
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in _rmtree_unsafe
          onerror(os.unlink, fullname, sys.exc_info())
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
      
      path = 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
      onerror = <function rmtree.<locals>.onerror at 0x000001D6C3E5C7B8>
      
          def _rmtree_unsafe(path, onerror):
              try:
                  if os.path.islink(path):
                      # symlinks to directories are forbidden, see bug #1669
                      raise OSError("Cannot call rmtree on a symbolic link")
              except OSError:
                  onerror(os.path.islink, path, sys.exc_info())
                  # can't continue even if onerror hook returns
                  return
              names = []
              try:
                  names = os.listdir(path)
              except OSError:
                  onerror(os.listdir, path, sys.exc_info())
              for name in names:
                  fullname = os.path.join(path, name)
                  try:
                      mode = os.lstat(fullname).st_mode
                  except OSError:
                      mode = 0
                  if stat.S_ISDIR(mode):
                      _rmtree_unsafe(fullname, onerror)
                  else:
                      try:
      >                   os.unlink(fullname)
      E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'
      
      c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError
      

        Attachments

          Activity

            People

            • Assignee:
              rohdesam Sam Rohde
              Reporter:
              bhulette Brian Hulette
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 50m
                1h 50m