[KUDU-2432] isolate race creating directory via dist_test.py

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: n/a
• Component/s: test
• Assignee: Todd Lipcon
• Reporter: Mike Percy
• Created: 08/May/18 01:36
• Updated: 14/May/20 18:01
• Resolved: 14/May/20 18:01

Description

When running dist_test.py I have been getting a 1% failure rate due to the following errors.

I am not sure if this is new or related to a single bad machine.

failed to download task files: WARNING 123 isolateserver(1484): Adding unknown file 7cf0792d18a9dbef867c9bce0c681b3def0510b6 to cache
WARNING 126 isolateserver(1490): Added back 1 unknown files
INFO 135 tools(106): Profiling: Section Setup took 0.045 seconds
INFO 164 tools(106): Profiling: Section GetIsolateds took 0.029 seconds
INFO 167 tools(106): Profiling: Section GetRest took 0.003 seconds
INFO 175 isolateserver(1365): 1 ( 227022kb) added
INFO 176 isolateserver(1369): 1642 ( 3864634kb) current
INFO 176 isolateserver(1372): 0 ( 0kb) removed
INFO 176 isolateserver(1375): 45627408kb free
INFO 176 tools(106): Profiling: Section CleanupTrimming took 0.009 seconds
INFO 177 isolateserver(1365): 1 ( 227022kb) added
INFO 177 isolateserver(1369): 1642 ( 3864634kb) current
INFO 177 isolateserver(1372): 0 ( 0kb) removed
INFO 177 isolateserver(1375): 45627408kb free
INFO 178 tools(106): Profiling: Section CleanupTrimming took 0.001 seconds
INFO 178 isolateserver(381): Waiting for all threads to die...
INFO 178 isolateserver(390): Done.
Traceback (most recent call last):
File "/swarming.client/isolateserver.py", line 2211, in <module>
sys.exit(main(sys.argv[1:]))
File "/swarming.client/isolateserver.py", line 2204, in main
return dispatcher.execute(OptionParserIsolateServer(), args)
File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute
return command(parser, args[1:])
File "/swarming.client/isolateserver.py", line 2064, in CMDdownload
require_command=False)
File "/swarming.client/isolateserver.py", line 1827, in fetch_isolated
create_directories(outdir, bundle.files)
File "/swarming.client/isolateserver.py", line 212, in create_directories
os.mkdir(os.path.join(base_directory, d))
OSError: [Errno 17] File exists: '/tmp/dist-test-task_gm4pM/build'