[MESOS-1655] ZooKeeperTest.LeaderDetectorTimeoutHandling is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.0
• Fix Version/s: 0.20.0
• Component/s: test

Environment

https://builds.apache.org/computer/ubuntu3/systemInfo

• Target Version/s: 0.20.0
• Sprint: Q3 Sprint 2
• Assignee: Yan Xu
• Reporter: Yan Xu
• Created: 30/Jul/14 20:53
• Updated: 01/Aug/14 18:48
• Resolved: 01/Aug/14 18:48

Description

[ RUN      ] ZooKeeperTest.LeaderDetectorTimeoutHandling
I0729 21:39:27.352937 17989 zookeeper_test_server.cpp:158] Started ZooKeeperTestServer on port 55004
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@716: Client environment:host.name=hemera
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-61-generic
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@725: Client environment:os.version=#93-Ubuntu SMP Fri May 2 21:31:50 UTC 2014
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@log_env@753: Client environment:user.dir=/x1/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/mesos-0.20.0/_build/src
2014-07-29 21:39:27,353:17989(0x2b3ae597d700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:55004 sessionTimeout=10000 watcher=0x2b3ae27a99e0 sessionId=0 sessionPasswd=<null> context=0x2b3b380291a0 flags=0
2014-07-29 21:39:27,365:17989(0x2b3d86260700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:55004]
2014-07-29 21:39:27,393:17989(0x2b3d86260700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:55004], sessionId=0x14784104ff50000, negotiated timeout=10000
I0729 21:39:27.401814 18028 group.cpp:313] Group process (group(13)@140.211.11.27:33282) connected to ZooKeeper
I0729 21:39:27.401831 18028 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I0729 21:39:27.401839 18028 group.cpp:385] Trying to create path '/test' in ZooKeeper
I0729 21:39:27.445768 18027 detector.cpp:138] Detected a new leader: (id='0')
I0729 21:39:27.447403 17989 zookeeper_test_server.cpp:122] Shutdown ZooKeeperTestServer on port 55004
2014-07-29 21:39:27,447:17989(0x2b3d86260700):ZOO_ERROR@handle_socket_error_msg@1721: Socket [127.0.0.1:55004] zk retcode=-4, errno=112(Host is down): failed while receiving a server response
I0729 21:39:27.457828 18016 group.cpp:418] Lost connection to ZooKeeper, attempting to reconnect ...
W0729 21:39:27.473925 18027 group.cpp:456] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=14784104ff50000) expiration
I0729 21:39:27.474212 18027 group.cpp:472] ZooKeeper session expired
I0729 21:39:27.474318 18026 detector.cpp:126] The current leader (id=0) is lost
I0729 21:39:27.474329 18026 detector.cpp:138] Detected a new leader: None
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@zookeeper_close@2522: Freeing zookeeper resources for sessionId=0x14784104ff50000

2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@716: Client environment:host.name=hemera
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@724: Client environment:os.arch=3.2.0-61-generic
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@725: Client environment:os.version=#93-Ubuntu SMP Fri May 2 21:31:50 UTC 2014
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@741: Client environment:user.home=/home/jenkins
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@log_env@753: Client environment:user.dir=/x1/jenkins/jenkins-slave/workspace/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/mesos-0.20.0/_build/src
2014-07-29 21:39:27,481:17989(0x2b3ae6f88700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=127.0.0.1:55004 sessionTimeout=10000 watcher=0x2b3ae27a99e0 sessionId=0 sessionPasswd=<null> context=0x2b3b6c00edc0 flags=0
2014-07-29 21:39:27,493:17989(0x2b3d85e4e700):ZOO_INFO@check_events@1703: initiated connection to server [127.0.0.1:55004]
I0729 21:39:27.510174 17989 zookeeper_test_server.cpp:158] Started ZooKeeperTestServer on port 55004
2014-07-29 21:39:27,545:17989(0x2b3d85e4e700):ZOO_INFO@check_events@1750: session establishment complete on server [127.0.0.1:55004], sessionId=0x1478410508c0000, negotiated timeout=10000
I0729 21:39:27.569658 18011 group.cpp:313] Group process (group(13)@140.211.11.27:33282) connected to ZooKeeper
I0729 21:39:27.569676 18011 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0729 21:39:27.569684 18011 group.cpp:385] Trying to create path '/test' in ZooKeeper
I0729 21:39:27.649694 18011 detector.cpp:138] Detected a new leader: (id='0')
../../src/tests/zookeeper_tests.cpp:252: Failure
Failed to wait 10secs for leader.get().get().cancelled()
2014-07-29 21:39:37,669:17989(0x2b3ae4f73f80):ZOO_INFO@zookeeper_close@2505: Closing zookeeper sessionId=0x1478410508c0000 to [127.0.0.1:55004]

I0729 21:39:37.698052 17989 zookeeper_test_server.cpp:122] Shutdown ZooKeeperTestServer on port 55004
[  FAILED  ] ZooKeeperTest.LeaderDetectorTimeoutHandling (10350 ms)

From https://builds.apache.org/job/Mesos-Trunk-Ubuntu-Build-In-Src-Set-JAVA_HOME/2017/consoleFull