[BEAM-10921] Interactive Runner Python unit tests are flaky on Windows

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-py-core
• Labels: currently-failing flake
• Assignee: Sam Rohde
• Reporter: Brian Hulette
• Created: 17/Sep/20 16:59
• Updated: 12/Apr/21 21:44
• Resolved: 12/Apr/21 21:44

Description

Over the past few days python unit tests have been failing frequently. The errors always seem to occur when cleaning up the interactive environment:

...........                                                              [100%]
================================== FAILURES ===================================
_ PipelineInstrumentTest.test_able_to_cache_intermediate_unbounded_source_pcollection _
[gw2] win32 -- Python 3.5.4 d:\a\beam\beam\sdks\python\target\.tox\py35-win\scripts\python.exe

self = <apache_beam.runners.interactive.pipeline_instrument_test.PipelineInstrumentTest testMethod=test_able_to_cache_intermediate_unbounded_source_pcollection>

    def setUp(self):
>     ie.new_env()

apache_beam\runners\interactive\pipeline_instrument_test.py:46: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam\runners\interactive\interactive_environment.py:117: in new_env
    _interactive_beam_env.cleanup()
apache_beam\runners\interactive\interactive_environment.py:273: in cleanup
    cache_manager.cleanup()
apache_beam\runners\interactive\caching\streaming_cache.py:391: in cleanup
    shutil.rmtree(self._cache_dir)
c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:494: in rmtree
    return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:384: in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:389: in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full'
onerror = <function rmtree.<locals>.onerror at 0x000001D6C3E5C7B8>

    def _rmtree_unsafe(path, onerror):
        try:
            if os.path.islink(path):
                # symlinks to directories are forbidden, see bug #1669
                raise OSError("Cannot call rmtree on a symbolic link")
        except OSError:
            onerror(os.path.islink, path, sys.exc_info())
            # can't continue even if onerror hook returns
            return
        names = []
        try:
            names = os.listdir(path)
        except OSError:
            onerror(os.listdir, path, sys.exc_info())
        for name in names:
            fullname = os.path.join(path, name)
            try:
                mode = os.lstat(fullname).st_mode
            except OSError:
                mode = 0
            if stat.S_ISDIR(mode):
                _rmtree_unsafe(fullname, onerror)
            else:
                try:
>                   os.unlink(fullname)
E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\\a\\beam\\beam\\sdks\\python\\target\\.tox\\py35-win\\tmp\\it-8vh2z7pi2021914046928\\full\\ac8879590f-2021876280456-2021876278608-2021914046928'

c:\hostedtoolcache\windows\python\3.5.4\x64\lib\shutil.py:387: PermissionError

Issue Links

• is duplicated by: BEAM-11209 Python Unit Tests (windows-latest, 3.7, py37)
• links to: GitHub Pull Request #12865
• links to: GitHub Pull Request #12866
• links to: GitHub Pull Request #13277
• links to: GitHub Pull Request #13335