[MESOS-229] mesos zookeeper group code fails to connect when pre-existing children of the group path are read-only

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: master
• Assignee: Benjamin Hindman
• Reporter: John Sirois
• Created: 08/Jul/12 23:54
• Updated: 22/May/13 01:24
• Resolved: 22/May/13 01:24

Description

We have a locked down zk tree as follows:
/ world:anyone:r
/home world:anyone:r
/home/mesos world:anyone:r,digest:mesos:XXX:cdrwa
...

This causes the mesos GroupProcess::connected code to fail as follows attempting to establish pre-existing path /home/mesos/prod/master for the master contender:

2012-07-08 23:25:36,322:60909(0x4d50e940):ZOO_INFO@check_events@1632: session establishment complete on server [10.40.93.122:2181], sessionId=0x43836278f888947, negotiated timeout=10000
2012-07-08 23:25:36,322:60909(0x4d50e940):ZOO_DEBUG@check_events@1638: Calling a watcher for a ZOO_SESSION_EVENT and the state=ZOO_CONNECTED_STATE
2012-07-08 23:25:36,322:60909(0x4dd0f940):ZOO_DEBUG@process_completions@1765: Calling a watcher for node [], type = -1 event=ZOO_SESSION_EVENT
I0708 23:25:36.322985 60928 detector.cpp:286] Master detector connected to ZooKeeper ...
I0708 23:25:36.323076 60928 detector.cpp:292] Authenticating to ZooKeeper using scheme 'digest'
2012-07-08 23:25:36,323:60909(0x49d07940):ZOO_DEBUG@send_last_auth_info@1265: Sending auth info request to 10.40.93.122:2181
I0708 23:25:36.329813 60937 webui_utils.cpp:49] Loading webui script at '/usr/local/share/mesos/webui/master/webui.py'
2012-07-08 23:25:39,656:60909(0x4d50e940):ZOO_DEBUG@zookeeper_process@1933: Processing AUTH_XID
2012-07-08 23:25:39,656:60909(0x4d50e940):ZOO_INFO@auth_completion_func@1198: Authentication scheme digest succeeded
2012-07-08 23:25:39,656:60909(0x4d50e940):ZOO_DEBUG@zookeeper_process@1983: Got ping response in 0 ms
I0708 23:25:39.657078 60928 detector.cpp:314] Trying to create znode '/home' in ZooKeeper
2012-07-08 23:25:39,657:60909(0x49d07940):ZOO_DEBUG@zoo_acreate@2503: Sending request xid=0x4ffa16f1 for path [/home] to 10.40.93.122:2181
2012-07-08 23:25:39,659:60909(0x4d50e940):ZOO_DEBUG@zookeeper_process@1989: Queueing asynchronous response
2012-07-08 23:25:39,659:60909(0x4dd0f940):ZOO_DEBUG@process_completions@1817: Calling COMPLETION_STRING for xid=0x4ffa16f1 rc=-102
Failed to create ZooKeeper znode: not authenticated (../../src/detector/detector.cpp:320)

The end observable behavior of GroupProcess::connected code should ideally ensure the path exists and attempt to create nodes if they do not exist. The current behavior is that GroupProcess::connected will refuse to use paths with read-only root nodes even if the leaf node it needs to manage a group is writable.

Issue Links

• relates to: MESOS-409 Master detector code should stat nodes before attempting to create