2020-03-20 04:12:19,360 INFO got task: {"isolate_hash": "f48f3144043e529db32f195f94316fec9ac5dd3c", "attempt": 0, "description": "hms_catalog-test.3", "task_id": "f48f3144043e529db32f195f94316fec9ac5dd3c.180", "artifact_archive_globs": ["build/*/test-logs/**/*"], "max_retries": 0, "timeout": 930, "job_id": "jenkins.1584676741.141402"} INFO:dist_test.slave:got task: {"isolate_hash": "f48f3144043e529db32f195f94316fec9ac5dd3c", "attempt": 0, "description": "hms_catalog-test.3", "task_id": "f48f3144043e529db32f195f94316fec9ac5dd3c.180", "artifact_archive_globs": ["build/*/test-logs/**/*"], "max_retries": 0, "timeout": 930, "job_id": "jenkins.1584676741.141402"} 2020-03-20 04:12:19,374 INFO Downloading files from isolate... INFO:dist_test.slave:Downloading files from isolate... 2020-03-20 04:12:19,374 INFO Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] INFO:dist_test.slave:Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] 2020-03-20 04:12:22,254 WARNING failed to download task files. 2 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: INFO 234 tools(106): Profiling: Section Setup took 0.145 seconds INFO 245 connectionpool(257): Starting new HTTP connection (1): isolateserver INFO 357 connectionpool(257): Starting new HTTP connection (2): isolateserver INFO 359 connectionpool(257): Starting new HTTP connection (3): isolateserver INFO 363 connectionpool(257): Starting new HTTP connection (4): isolateserver INFO 366 connectionpool(257): Starting new HTTP connection (5): isolateserver INFO 558 tools(106): Profiling: Section GetIsolateds took 0.322 seconds INFO 579 isolateserver(1843): Retrieving remaining files (7 of them)... ERROR 1119 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 1209 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 1950 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 2115 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 2560 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer INFO 2561 connectionpool(257): Starting new HTTP connection (6): isolateserver ERROR 2773 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer INFO 2800 tools(106): Profiling: Section GetRest took 2.242 seconds INFO 2827 isolateserver(1365): 7 ( 29621kb) added INFO 2829 isolateserver(1369): 5521 ( 7672329kb) current INFO 2829 isolateserver(1372): 0 ( 0kb) removed INFO 2829 isolateserver(1375): 40013000kb free INFO 2829 tools(106): Profiling: Section CleanupTrimming took 0.029 seconds INFO 2830 isolateserver(1365): 7 ( 29621kb) added INFO 2832 isolateserver(1369): 5521 ( 7672329kb) current INFO 2832 isolateserver(1372): 0 ( 0kb) removed INFO 2832 isolateserver(1375): 40013000kb free INFO 2832 tools(106): Profiling: Section CleanupTrimming took 0.003 seconds INFO 2832 isolateserver(381): Waiting for all threads to die... INFO 2834 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1850, in fetch_isolated digest = fetch_queue.wait(remaining) File "/swarming.client/isolateserver.py", line 744, in wait digest = self._channel.pull() File "/swarming.client/utils/threading_utils.py", line 377, in _task_executer result = func(*args, **kwargs) File "/swarming.client/isolateserver.py", line 597, in fetch sink(verifier.run()) File "/swarming.client/isolateserver.py", line 1417, in write size = file_write(path, content) File "/swarming.client/isolateserver.py", line 144, in file_write for d in content_generator: File "/swarming.client/isolateserver.py", line 792, in run for chunk in self.stream: File "/swarming.client/isolateserver.py", line 174, in zip_decompress for chunk in content_generator: File "/swarming.client/isolateserver.py", line 112, in stream_read data = stream.read(chunk_size) File "/swarming.client/utils/net.py", line 638, in read data = self._stream.read() if size is None else self._stream.read(size) File "/swarming.client/third_party/requests/packages/urllib3/response.py", line 174, in read data = self._fp.read(amt) File "/usr/lib/python2.7/httplib.py", line 578, in read return self._read_chunked(amt) File "/usr/lib/python2.7/httplib.py", line 642, in _read_chunked value.append(self._safe_read(amt)) File "/usr/lib/python2.7/httplib.py", line 693, in _safe_read chunk = self.fp.read(min(amt, MAXAMOUNT)) File "/usr/lib/python2.7/socket.py", line 380, in read data = self._sock.recv(left) socket.error: [Errno 104] Connection reset by peer WARNING:dist_test.slave:failed to download task files. 2 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: INFO 234 tools(106): Profiling: Section Setup took 0.145 seconds INFO 245 connectionpool(257): Starting new HTTP connection (1): isolateserver INFO 357 connectionpool(257): Starting new HTTP connection (2): isolateserver INFO 359 connectionpool(257): Starting new HTTP connection (3): isolateserver INFO 363 connectionpool(257): Starting new HTTP connection (4): isolateserver INFO 366 connectionpool(257): Starting new HTTP connection (5): isolateserver INFO 558 tools(106): Profiling: Section GetIsolateds took 0.322 seconds INFO 579 isolateserver(1843): Retrieving remaining files (7 of them)... ERROR 1119 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 1209 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 1950 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 2115 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer ERROR 2560 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer INFO 2561 connectionpool(257): Starting new HTTP connection (6): isolateserver ERROR 2773 isolateserver(599): Failed to fetch baa9a30cbb3949d820be6ab9fe839a68b111a2f1: [Errno 104] Connection reset by peer INFO 2800 tools(106): Profiling: Section GetRest took 2.242 seconds INFO 2827 isolateserver(1365): 7 ( 29621kb) added INFO 2829 isolateserver(1369): 5521 ( 7672329kb) current INFO 2829 isolateserver(1372): 0 ( 0kb) removed INFO 2829 isolateserver(1375): 40013000kb free INFO 2829 tools(106): Profiling: Section CleanupTrimming took 0.029 seconds INFO 2830 isolateserver(1365): 7 ( 29621kb) added INFO 2832 isolateserver(1369): 5521 ( 7672329kb) current INFO 2832 isolateserver(1372): 0 ( 0kb) removed INFO 2832 isolateserver(1375): 40013000kb free INFO 2832 tools(106): Profiling: Section CleanupTrimming took 0.003 seconds INFO 2832 isolateserver(381): Waiting for all threads to die... INFO 2834 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1850, in fetch_isolated digest = fetch_queue.wait(remaining) File "/swarming.client/isolateserver.py", line 744, in wait digest = self._channel.pull() File "/swarming.client/utils/threading_utils.py", line 377, in _task_executer result = func(*args, **kwargs) File "/swarming.client/isolateserver.py", line 597, in fetch sink(verifier.run()) File "/swarming.client/isolateserver.py", line 1417, in write size = file_write(path, content) File "/swarming.client/isolateserver.py", line 144, in file_write for d in content_generator: File "/swarming.client/isolateserver.py", line 792, in run for chunk in self.stream: File "/swarming.client/isolateserver.py", line 174, in zip_decompress for chunk in content_generator: File "/swarming.client/isolateserver.py", line 112, in stream_read data = stream.read(chunk_size) File "/swarming.client/utils/net.py", line 638, in read data = self._stream.read() if size is None else self._stream.read(size) File "/swarming.client/third_party/requests/packages/urllib3/response.py", line 174, in read data = self._fp.read(amt) File "/usr/lib/python2.7/httplib.py", line 578, in read return self._read_chunked(amt) File "/usr/lib/python2.7/httplib.py", line 642, in _read_chunked value.append(self._safe_read(amt)) File "/usr/lib/python2.7/httplib.py", line 693, in _safe_read chunk = self.fp.read(min(amt, MAXAMOUNT)) File "/usr/lib/python2.7/socket.py", line 380, in read data = self._sock.recv(left) socket.error: [Errno 104] Connection reset by peer 2020-03-20 04:12:27,259 INFO Downloading files from isolate... INFO:dist_test.slave:Downloading files from isolate... 2020-03-20 04:12:27,260 INFO Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] INFO:dist_test.slave:Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] 2020-03-20 04:12:28,527 WARNING failed to download task files. 1 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: INFO 223 tools(106): Profiling: Section Setup took 0.132 seconds INFO 251 connectionpool(257): Starting new HTTP connection (1): isolateserver INFO 344 tools(106): Profiling: Section GetIsolateds took 0.120 seconds INFO 351 tools(106): Profiling: Section GetRest took 0.006 seconds INFO 404 isolateserver(1365): 0 ( 0kb) added INFO 407 isolateserver(1369): 5521 ( 7672329kb) current INFO 408 isolateserver(1372): 0 ( 0kb) removed INFO 408 isolateserver(1375): 40008996kb free INFO 408 tools(106): Profiling: Section CleanupTrimming took 0.057 seconds INFO 411 isolateserver(1365): 0 ( 0kb) added INFO 413 isolateserver(1369): 5521 ( 7672329kb) current INFO 413 isolateserver(1372): 0 ( 0kb) removed INFO 413 isolateserver(1375): 40005836kb free INFO 413 tools(106): Profiling: Section CleanupTrimming took 0.005 seconds INFO 414 isolateserver(381): Waiting for all threads to die... INFO 1222 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1827, in fetch_isolated create_directories(outdir, bundle.files) File "/swarming.client/isolateserver.py", line 212, in create_directories os.mkdir(os.path.join(base_directory, d)) OSError: [Errno 17] File exists: '/tmp/dist-test-taskLm7YLa/build' WARNING:dist_test.slave:failed to download task files. 1 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: INFO 223 tools(106): Profiling: Section Setup took 0.132 seconds INFO 251 connectionpool(257): Starting new HTTP connection (1): isolateserver INFO 344 tools(106): Profiling: Section GetIsolateds took 0.120 seconds INFO 351 tools(106): Profiling: Section GetRest took 0.006 seconds INFO 404 isolateserver(1365): 0 ( 0kb) added INFO 407 isolateserver(1369): 5521 ( 7672329kb) current INFO 408 isolateserver(1372): 0 ( 0kb) removed INFO 408 isolateserver(1375): 40008996kb free INFO 408 tools(106): Profiling: Section CleanupTrimming took 0.057 seconds INFO 411 isolateserver(1365): 0 ( 0kb) added INFO 413 isolateserver(1369): 5521 ( 7672329kb) current INFO 413 isolateserver(1372): 0 ( 0kb) removed INFO 413 isolateserver(1375): 40005836kb free INFO 413 tools(106): Profiling: Section CleanupTrimming took 0.005 seconds INFO 414 isolateserver(381): Waiting for all threads to die... INFO 1222 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1827, in fetch_isolated create_directories(outdir, bundle.files) File "/swarming.client/isolateserver.py", line 212, in create_directories os.mkdir(os.path.join(base_directory, d)) OSError: [Errno 17] File exists: '/tmp/dist-test-taskLm7YLa/build' 2020-03-20 04:12:33,533 INFO Downloading files from isolate... INFO:dist_test.slave:Downloading files from isolate... 2020-03-20 04:12:33,533 INFO Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] INFO:dist_test.slave:Running command: ['/swarming.client/isolateserver.py', 'download', '--isolate-server=http://isolateserver:4242/', '--cache=/tmp/isolate-cache.0', '--verbose', '-i', u'f48f3144043e529db32f195f94316fec9ac5dd3c', '--target', '/tmp/dist-test-taskLm7YLa'] 2020-03-20 04:12:33,921 WARNING failed to download task files. 0 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: WARNING 209 isolateserver(1484): Adding unknown file baa9a30cbb3949d820be6ab9fe839a68b111a2f1 to cache WARNING 224 isolateserver(1490): Added back 1 unknown files INFO 250 tools(106): Profiling: Section Setup took 0.155 seconds INFO 309 tools(106): Profiling: Section GetIsolateds took 0.058 seconds INFO 313 tools(106): Profiling: Section GetRest took 0.003 seconds INFO 339 isolateserver(1365): 1 ( 91914kb) added INFO 341 isolateserver(1369): 5522 ( 7764243kb) current INFO 341 isolateserver(1372): 0 ( 0kb) removed INFO 341 isolateserver(1375): 39921044kb free INFO 341 tools(106): Profiling: Section CleanupTrimming took 0.028 seconds INFO 342 isolateserver(1365): 1 ( 91914kb) added INFO 344 isolateserver(1369): 5522 ( 7764243kb) current INFO 344 isolateserver(1372): 0 ( 0kb) removed INFO 344 isolateserver(1375): 39921044kb free INFO 344 tools(106): Profiling: Section CleanupTrimming took 0.003 seconds INFO 344 isolateserver(381): Waiting for all threads to die... INFO 344 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1827, in fetch_isolated create_directories(outdir, bundle.files) File "/swarming.client/isolateserver.py", line 212, in create_directories os.mkdir(os.path.join(base_directory, d)) OSError: [Errno 17] File exists: '/tmp/dist-test-taskLm7YLa/build' WARNING:dist_test.slave:failed to download task files. 0 tries remaining Traceback (most recent call last): File "/dist_test/infra/slave.py", line 211, in download_task_files_with_retries return self.download_task_files(*args, **kwargs) File "/dist_test/infra/slave.py", line 189, in download_task_files raise Exception("failed to download task files: %s" % stderr) Exception: failed to download task files: WARNING 209 isolateserver(1484): Adding unknown file baa9a30cbb3949d820be6ab9fe839a68b111a2f1 to cache WARNING 224 isolateserver(1490): Added back 1 unknown files INFO 250 tools(106): Profiling: Section Setup took 0.155 seconds INFO 309 tools(106): Profiling: Section GetIsolateds took 0.058 seconds INFO 313 tools(106): Profiling: Section GetRest took 0.003 seconds INFO 339 isolateserver(1365): 1 ( 91914kb) added INFO 341 isolateserver(1369): 5522 ( 7764243kb) current INFO 341 isolateserver(1372): 0 ( 0kb) removed INFO 341 isolateserver(1375): 39921044kb free INFO 341 tools(106): Profiling: Section CleanupTrimming took 0.028 seconds INFO 342 isolateserver(1365): 1 ( 91914kb) added INFO 344 isolateserver(1369): 5522 ( 7764243kb) current INFO 344 isolateserver(1372): 0 ( 0kb) removed INFO 344 isolateserver(1375): 39921044kb free INFO 344 tools(106): Profiling: Section CleanupTrimming took 0.003 seconds INFO 344 isolateserver(381): Waiting for all threads to die... INFO 344 isolateserver(390): Done. Traceback (most recent call last): File "/swarming.client/isolateserver.py", line 2211, in sys.exit(main(sys.argv[1:])) File "/swarming.client/isolateserver.py", line 2204, in main return dispatcher.execute(OptionParserIsolateServer(), args) File "/swarming.client/third_party/depot_tools/subcommand.py", line 242, in execute return command(parser, args[1:]) File "/swarming.client/isolateserver.py", line 2064, in CMDdownload require_command=False) File "/swarming.client/isolateserver.py", line 1827, in fetch_isolated create_directories(outdir, bundle.files) File "/swarming.client/isolateserver.py", line 212, in create_directories os.mkdir(os.path.join(base_directory, d)) OSError: [Errno 17] File exists: '/tmp/dist-test-taskLm7YLa/build' 2020-03-20 04:12:34,215 INFO Removing test directory /tmp/dist-test-taskLm7YLa INFO:dist_test.slave:Removing test directory /tmp/dist-test-taskLm7YLa