Major Description
The following failure message occured in a test cluster at Twitter:
// We fail all non-OK return codes except ZNODEEXISTS (since that
// means the path we were trying to create exists) and ZNOAUTH
// (since it's possible that the ACLs on 'dirname(url.path)' don't
// allow us to create a child znode but we are allowed to create
// children of 'url.path' itself, which will be determined below
// if we are contending). Note that it's also possible we got back
// a ZNONODE because we could not create one of the intermediate
// znodes (in which case we'll abort in the 'else' below since
// ZNONODE is non-retryable). TODO(benh): Need to check that we
// also can put a watch on the children of 'url.path'.
if (code != ZOK && code != ZNODEEXISTS && code != ZNOAUTH)
It's interesting that there was a delay before the slave crashed:
F0506 21:59:46.769142 54944 detector.cpp:315] Failed to create '/home/mesos/test/master' in ZooKeeper: invalid zhandle state
-
-
- Check failure stack trace: ***
I0506 21:59:47.398509 54940 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877344969-mesos-meta_slave_4-7-045d344f-1737-4901-9720-6663aee94840_tag
_c6f62383-28d1-42c2-aa65-15f9ec42db57
I0506 21:59:46.538936 54936 cgroups_isolator.cpp:804] Executor thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22 of framework 201103282247-0000000019-0000 terminated with status 0
I0506 22:00:01.329169 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22 of framework 201103282247-0000000019-0000
I0506 21:59:47.576560 54948 slave.cpp:721] Got assigned task 1367877572534-mesos-meta_slave_2-4-3b3755d9-9435-4480-9068-d851234cb91d for framework 201103282247-0000000019-0000
W0506 21:59:47.576581 54941 monitor.cpp:167] Failed to collect resource usage for executor 'thermos-1367877382194-mesos-meta_slave_14-19-cb386113-baa0-4474-a8de-7e39564e5bfc' of framework '201103282247-000000001
9-0000': 1
W0506 21:59:51.089640 54939 cgroups.cpp:1261] Unable to freeze /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877355794-mesos-meta_slave_41-2-6f44eb1d-9d77-4201-a3dd-376abb0df8d0_tag_b
26ba555-a6bd-4448-a5b4-439beb442820 within 51 attempts
W0506 21:59:51.090683 54943 cgroups.cpp:1261] Unable to freeze /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877382194-mesos-meta_slave_14-19-cb386113-baa0-4474-a8de-7e39564e5bfc_tag_
6ce11d7c-ab7c-4e25-9d9d-7f37fd04c3f4 within 51 attempts
I0506 21:59:47.518194 54946 status_update_manager.cpp:359] Received status update acknowledgement c7701021-7eac-4711-96ac-093871462e44 for task 1367877461237-mesos-meta_slave_33-21-e1474932-1a9b-4adf-acbf-9f8a48
f50c4a of framework 201103282247-0000000019-0000
W0506 22:00:01.439838 54941 monitor.cpp:167] Failed to collect resource usage for executor 'thermos-1367877344969-mesos-meta_slave_4-7-045d344f-1737-4901-9720-6663aee94840' of framework '201103282247-0000000019-
0000': 1
I0506 22:00:01.468596 54946 status_update_manager.cpp:359] Received status update acknowledgement 76ed08a3-1f8a-4eb2-8d2a-28de822ee370 for task 1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767ac
b0d90 of framework 201103282247-0000000019-0000
I0506 22:00:01.580569 54946 status_update_manager.cpp:359] Received status update acknowledgement 146b821e-ed93-4059-8971-b3c73a7d02ed for task 1367877517296-mesos-meta_slave_22-4-e2101290-0a8e-4de1-a5fa-d754a82
c86f6 of framework 201103282247-0000000019-0000
W0506 22:00:01.580670 54946 status_update_manager.cpp:382] Ignoring unexpected status update acknowledgment 146b821e-ed93-4059-8971-b3c73a7d02ed for task 1367877517296-mesos-meta_slave_22-4-e2101290-0a8e-4de1-a5
fa-d754a82c86f6 of framework 201103282247-0000000019-0000
I0506 22:00:01.888521 54936 cgroups_isolator.cpp:804] Executor thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e1c of framework 201103282247-0000000019-0000 terminated with status 0
I0506 22:00:06.876126 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e1c of framework 201103282247-0000000019-0000
W0506 22:00:01.917793 54946 status_update_manager.cpp:432] Resending status update TASK_LOST (UUID: e7e02c18-abee-4781-9e62-70e0475b9fa3) for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400
f9dd7c22 of framework 201103282247-0000000019-0000
I0506 22:00:08.930923 54946 status_update_manager.cpp:335] Forwarding status update TASK_LOST (UUID: e7e02c18-abee-4781-9e62-70e0475b9fa3) for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f40
0f9dd7c22 of framework 201103282247-0000000019-0000 to master@10.34.128.115:5050
I0506 22:00:02.215196 54949 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877382194-mesos-meta_slave_14-19-cb386113-baa0-4474-a8de-7e39564e5bfc
_tag_6ce11d7c-ab7c-4e25-9d9d-7f37fd04c3f4
I0506 22:00:02.243545 54945 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877355794-mesos-meta_slave_41-2-6f44eb1d-9d77-4201-a3dd-376abb0df8d0_
tag_b26ba555-a6bd-4448-a5b4-439beb442820
I0506 22:00:09.717344 54949 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877382194-mesos-meta_slave_14-19-cb386113-baa0-4474-a8de-7e39564e5bfc_t
ag_6ce11d7c-ab7c-4e25-9d9d-7f37fd04c3f4
W0506 22:00:01.917774 54941 monitor.cpp:167] Failed to collect resource usage for executor 'thermos-1367877355794-mesos-meta_slave_41-2-6f44eb1d-9d77-4201-a3dd-376abb0df8d0' of framework '201103282247-0000000019
-0000': 1
I0506 22:00:06.931675 54936 cgroups_isolator.cpp:804] Executor thermos-1367877427667-mesos-meta_slave_37-7-7cf097a4-c9da-4972-83b9-e2aab5588019 of framework 201103282247-0000000019-0000 terminated with status 0
I0506 22:00:06.960703 54934 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e
1c_tag_720a3ca1-ea57-421f-9263-d47e347b866b
I0506 22:00:02.157475 54948 slave.cpp:514] Successfully attached file '/var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/thermos-1367877538347-mesos-me
ta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16/runs/ed262fff-5bc0-412b-8c86-799354704486'
I0506 22:00:02.243666 54937 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c2
2_tag_2407f4f8-2ad7-4415-abdd-016d5ab8e20d
I0506 22:00:09.717412 54945 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877355794-mesos-meta_slave_41-2-6f44eb1d-9d77-4201-a3dd-376abb0df8d0_ta
g_b26ba555-a6bd-4448-a5b4-439beb442820
I0506 22:00:09.766968 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877427667-mesos-meta_slave_37-7-7cf097a4-c9da-4972-83b9-e2aab5588019 of framework 201103282247-0000000019-0000
I0506 22:00:09.794438 54948 slave.cpp:2031] Executor 'thermos-1367877283998-mesos-meta_slave_25-5-b56d573d-e21c-446d-be54-e69d558f9bb2' of framework 201103282247-0000000019-0000 has exited with status '0'
I0506 22:00:10.080941 54936 cgroups_isolator.cpp:804] Executor thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0d90 of framework 201103282247-0000000019-0000 terminated with status 0
I0506 22:00:10.081022 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0d90 of framework 201103282247-0000000019-0000
I0506 22:00:10.138010 54948 slave.cpp:2166] Cleaning up executor 'thermos-1367877283998-mesos-meta_slave_25-5-b56d573d-e21c-446d-be54-e69d558f9bb2' of framework 201103282247-0000000019-0000
@ 0x7f050768dbcd google::LogMessage::Fail()
I0506 22:00:10.421581 54936 cgroups_isolator.cpp:804] Executor thermos-1367877461237-mesos-meta_slave_33-21-e1474932-1a9b-4adf-acbf-9f8a48f50c4a of framework 201103282247-0000000019-0000 terminated with status 0
@ 0x7f0507693837 google::LogMessage::SendToLog()
I0506 22:00:21.140630 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877461237-mesos-meta_slave_33-21-e1474932-1a9b-4adf-acbf-9f8a48f50c4a of framework 201103282247-0000000019-0000
I0506 22:00:10.591079 54948 slave.cpp:819] Launching task 1367877549319-mesos-meta_slave_42-19-269da040-c052-42e0-ac7e-d365c0562d3a for framework 201103282247-0000000019-0000
I0506 22:00:10.703546 54934 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0d9
0_tag_e0d3ee4d-e89c-4892-a46d-f5a85854b3a0
I0506 22:00:11.635795 54935 gc.cpp:143] Deleted '/var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/gc-d8f1a18a-8527-4452-99d7-a2da4c3c2272/runs/c562a33
e-6870-47d8-ae53-8faec70ad328'
W0506 22:00:15.583375 54942 cgroups.cpp:1261] Unable to freeze /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22_tag_2
407f4f8-2ad7-4415-abdd-016d5ab8e20d within 51 attempts
W0506 22:00:15.596472 54941 cgroups.cpp:1261] Unable to freeze /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e1c_tag_
720a3ca1-ea57-421f-9263-d47e347b866b within 51 attempts
W0506 22:00:19.717893 54946 status_update_manager.cpp:432] Resending status update TASK_LOST (UUID: e7e02c18-abee-4781-9e62-70e0475b9fa3) for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400
f9dd7c22 of framework 201103282247-0000000019-0000
I0506 22:00:10.453220 54949 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877427667-mesos-meta_slave_37-7-7cf097a4-c9da-4972-83b9-e2aab558801
9_tag_9f6b6105-0a40-4e8f-8044-19a09b186984
I0506 22:00:21.141474 54946 status_update_manager.cpp:335] Forwarding status update TASK_LOST (UUID: e7e02c18-abee-4781-9e62-70e0475b9fa3) for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f40
0f9dd7c22 of framework 201103282247-0000000019-0000 to master@10.34.128.115:5050
I0506 22:00:21.143030 54948 paths.hpp:302] Created executor directory '/var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/thermos-1367877549319-mesos-me
ta_slave_42-19-269da040-c052-42e0-ac7e-d365c0562d3a/runs/bd162f3e-c45a-44a9-ae10-25914c460689'
I0506 22:00:21.256687 54948 slave.cpp:930] Queuing task '1367877549319-mesos-meta_slave_42-19-269da040-c052-42e0-ac7e-d365c0562d3a' for executor thermos-1367877549319-mesos-meta_slave_42-19-269da040-c052-42e0-ac
7e-d365c0562d3a of framework '201103282247-0000000019-0000
I0506 22:00:21.256707 54936 cgroups_isolator.cpp:804] Executor thermos-1367877408255-mesos-meta_slave_0-24-f4400b8c-4ba0-46db-b878-ddb275599233 of framework 201103282247-0000000019-0000 terminated with status 0
I0506 22:00:21.256798 54948 slave.cpp:1789] Sending acknowledgement for status update TASK_LOST (UUID: e7e02c18-abee-4781-9e62-70e0475b9fa3) for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f
400f9dd7c22 of framework 201103282247-0000000019-0000 to executor(1)@10.34.124.132:55886
I0506 22:00:21.256815 54936 cgroups_isolator.cpp:620] Killing executor thermos-1367877408255-mesos-meta_slave_0-24-f4400b8c-4ba0-46db-b878-ddb275599233 of framework 201103282247-0000000019-0000
@ 0x7f050768f47c google::LogMessage::Flush()
I0506 22:00:21.377487 54935 gc.cpp:134] Deleting /var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/gc-d8f1a18a-8527-4452-99d7-a2da4c3c2272
I0506 22:00:21.377707 54935 gc.cpp:143] Deleted '/var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/gc-d8f1a18a-8527-4452-99d7-a2da4c3c2272'
@ 0x7f050768f6e6 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f050741063d mesos::internal::ZooKeeperMasterDetectorProcess::connected()
@ 0x7f05074111d8 std::tr1::_Function_handler<>::_M_invoke()
@ 0x7f0507413c84 std::tr1::_Function_handler<>::_M_invoke()
@ 0x7f050758d99a process::ProcessManager::resume()
@ 0x7f050758e9af process::schedule()
@ 0x7f0506d2773d start_thread
I0506 22:00:21.487545 54940 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0
d90_tag_e0d3ee4d-e89c-4892-a46d-f5a85854b3a0 after 3 attempts
I0506 22:00:21.487633 54942 cgroups.cpp:1214] Successfully froze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877427667-mesos-meta_slave_37-7-7cf097a4-c9da-4972-83b9-e2aab5588
019_tag_9f6b6105-0a40-4e8f-8044-19a09b186984 after 3 attempts
I0506 22:00:21.519721 54935 gc.cpp:134] Deleting /var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/thermos-1367685279171-mesos-meta_slave_14-0-a67ae3d4
-3e67-4a3a-ba85-ec66ca255db8/runs/76e96368-7692-4d9b-a717-e27fe7873f8b
I0506 22:00:21.519790 54941 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877461237-mesos-meta_slave_33-21-e1474932-1a9b-4adf-acbf-9f8a48f50c
4a_tag_53fc2cf1-8f88-4913-9e87-d865dbff4cdf
I0506 22:00:21.577153 54946 status_update_manager.cpp:359] Received status update acknowledgement e7e02c18-abee-4781-9e62-70e0475b9fa3 for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9d
d7c22 of framework 201103282247-0000000019-0000
I0506 22:00:21.577270 54948 slave.cpp:721] Got assigned task 1367877588667-mesos-meta_slave_33-39-79414744-8178-48e5-98df-eec82f8091f0 for framework 201103282247-0000000019-0000
I0506 22:00:21.674876 54938 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e1c
_tag_720a3ca1-ea57-421f-9263-d47e347b866b
I0506 22:00:21.828469 54936 cgroups_isolator.cpp:520] Launching thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16 (./thermos_executor) in /var/lib/mesos/slaves/201304262233-1937777
162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16/runs/ed262fff-5bc0-412b-8c86-799354704486 with resources cpus=
0.25; mem=128 for framework 201103282247-0000000019-0000 in cgroup mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16_tag_ed262
fff-5bc0-412b-8c86-799354704486
I0506 22:00:21.828743 54949 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22_
tag_2407f4f8-2ad7-4415-abdd-016d5ab8e20d
I0506 22:00:21.828740 54937 cgroups.cpp:1175] Trying to freeze cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877408255-mesos-meta_slave_0-24-f4400b8c-4ba0-46db-b878-ddb27559923
3_tag_4b84157b-e9c6-43e8-84f9-9ad1467411a7
I0506 22:00:22.244297 54946 status_update_manager.cpp:480] Cleaning up status update stream for task 1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22 of framework 201103282247-0000000019-
0000
I0506 22:00:22.244469 54938 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877395272-mesos-meta_slave_32-42-b55e4f5b-6ee1-4785-bd49-0c38fcec5e1c_t
ag_720a3ca1-ea57-421f-9263-d47e347b866b
I0506 22:00:22.244581 54949 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877474973-mesos-meta_slave_9-28-45a3d2d8-004c-4724-9590-f400f9dd7c22_ta
g_2407f4f8-2ad7-4415-abdd-016d5ab8e20d
I0506 22:00:22.246649 54936 cgroups_isolator.cpp:655] Changing cgroup controls for executor thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16 of framework 201103282247-0000000019-0
000 with resources cpus=0.25; mem=128
I0506 22:00:22.247011 54936 cgroups_isolator.cpp:839] Updated 'cpu.shares' to 256 for executor thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16 of framework 201103282247-000000001
9-0000
I0506 22:00:22.247328 54936 cgroups_isolator.cpp:977] Updated 'memory.limit_in_bytes' to 134217728 for executor thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16 of framework 201103282247-0000000019-0000
I0506 22:00:22.622454 54948 http.cpp:279] HTTP request for '/slave(1)/stats.json'
I0506 22:00:23.052145 54947 cgroups.cpp:1190] Trying to thaw cgroup /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0d90_tag_e0d3ee4d-e89c-4892-a46d-f5a85854b3a0
I0506 22:00:23.052280 54947 cgroups.cpp:1298] Successfully thawed /cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1367877449740-mesos-meta_slave_4-34-b1914d5b-c474-4f01-b16d-44767acb0d90_tag_e0d3ee4d-e89c-4892-a46d-f5a85854b3a0
I0506 22:00:23.283198 54936 cgroups_isolator.cpp:1003] Started listening for OOM events for executor thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16 of framework 201103282247-0000000019-0000
I0506 22:00:23.296195 54936 cgroups_isolator.cpp:550] Forked executor at = 20045
@ 0x7f050570bf6d clone
Fetching resources into '/var/lib/mesos/slaves/201304262233-1937777162-5050-9099-380/frameworks/201103282247-0000000019-0000/executors/thermos-1367877538347-mesos-meta_slave_25-5-51502091-a395-43d9-9d56-594895d16c16/runs/ed262fff-5bc0-412b-8c86-799354704486'
Fetching resource '/usr/local/bin/thermos_executor'
Copying resource from '/usr/local/bin/thermos_executor' to .
/usr/local/bin/mesos-slave.sh: line 101: 54931 Aborted (core dumped) /usr/local/sbin/mesos-slave --port=5051 --resources="${MESOS_RESOURCES}" --attributes="${MESOS_ATTRIBUTES}" --master="${master_zoo_url}" --log_dir="${log_dir}" ${WORK_DIR} ${CGROUPS_ISOLATION} "$@"
Slave Exit Status: 134
I0506 22:01:17.601867 20405 main.cpp:124] Creating "cgroups" isolator
I0506 22:01:17.627717 20405 main.cpp:132] Build: 2013-04-26 19:49:25 by bmahler
I0506 22:01:17.627739 20405 main.cpp:133] Starting Mesos slave
- Check failure stack trace: ***
-