[MESOS-774] FaultToleranceTest.MasterFailover test is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0
• Assignee: Vinod Kone
• Reporter: Vinod Kone
• Created: 25/Oct/13 00:11
• Updated: 25/Oct/13 01:33
• Resolved: 25/Oct/13 01:33

Description

[ RUN ] FaultToleranceTest.MasterFailover
I1024 16:00:03.252568 5250 master.cpp:284] Master started on 127.0.0.1:48588
I1024 16:00:03.252671 5250 master.cpp:299] Master ID: 201310241600-16777343-48588-5229
I1024 16:00:03.252744 5250 master.cpp:302] Master only allowing authenticated frameworks to register!
I1024 16:00:03.253101 5250 master.cpp:697] Elected as master!
I1024 16:00:03.253240 5249 sched.cpp:195] New master at master@127.0.0.1:48588
I1024 16:00:03.253314 5249 sched.cpp:281] Authenticating with master master@127.0.0.1:48588
I1024 16:00:03.253593 5249 hierarchical_allocator_process.hpp:302] Initializing hierarchical allocator process with master : master@127.0.0.1:48588
I1024 16:00:03.253643 5249 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1024 16:00:03.253679 5249 authenticatee.hpp:124] Creating new client SASL connection
I1024 16:00:03.253947 5250 master.cpp:1702] Authenticating framework at scheduler(72)@127.0.0.1:48588
I1024 16:00:03.254154 5250 authenticator.hpp:140] Creating new server SASL connection
I1024 16:00:03.254297 5250 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5
I1024 16:00:03.254386 5250 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5'
I1024 16:00:03.254492 5250 authenticator.hpp:243] Received SASL authentication start
I1024 16:00:03.254603 5250 authenticator.hpp:325] Authentication requires more steps
I1024 16:00:03.254714 5249 authenticatee.hpp:258] Received SASL authentication step
I1024 16:00:03.254806 5250 authenticator.hpp:271] Received SASL authentication step
I1024 16:00:03.254927 5250 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1024 16:00:03.255005 5250 auxprop.cpp:153] Looking up auxiliary property '*userPassword'
I1024 16:00:03.255079 5250 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1024 16:00:03.255156 5250 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1024 16:00:03.255225 5250 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1024 16:00:03.255292 5250 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1024 16:00:03.255368 5250 authenticator.hpp:317] Authentication success
I1024 16:00:03.255468 5249 authenticatee.hpp:298] Authentication success
I1024 16:00:03.255511 5249 sched.cpp:326] Successfully authenticated with master master@127.0.0.1:48588
I1024 16:00:03.258111 5251 master.cpp:722] Queuing up registration request from scheduler(72)@127.0.0.1:48588 because authentication is still in progress
I1024 16:00:03.258453 5250 master.cpp:1742] Successfully authenticated framework at scheduler(72)@127.0.0.1:48588
I1024 16:00:03.258750 5250 master.cpp:755] Received registration request from scheduler(72)@127.0.0.1:48588
I1024 16:00:03.259048 5250 master.cpp:773] Registering framework 201310241600-16777343-48588-5229-0000 at scheduler(72)@127.0.0.1:48588
I1024 16:00:03.259323 5250 hierarchical_allocator_process.hpp:332] Added framework 201310241600-16777343-48588-5229-0000
I1024 16:00:03.259516 5250 hierarchical_allocator_process.hpp:726] No resources available to allocate!
I1024 16:00:03.259711 5250 hierarchical_allocator_process.hpp:688] Performed allocation for 0 slaves in 194149ns
I1024 16:00:03.260061 5229 master.cpp:557] Master terminating
I1024 16:00:03.260085 5229 master.cpp:209] Shutting down master
I1024 16:00:03.260682 5252 master.cpp:284] Master started on 127.0.0.1:48588
I1024 16:00:03.260710 5252 master.cpp:299] Master ID: 201310241600-16777343-48588-5229
I1024 16:00:03.260717 5252 master.cpp:302] Master only allowing authenticated frameworks to register!
I1024 16:00:03.260984 5252 master.cpp:697] Elected as master!
I1024 16:00:03.261073 5252 hierarchical_allocator_process.hpp:302] Initializing hierarchical allocator process with master : master@127.0.0.1:48588
I1024 16:00:03.261101 5252 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1024 16:00:03.261325 5249 sched.cpp:365] Framework registered with 201310241600-16777343-48588-5229-0000
I1024 16:00:03.263921 5249 sched.cpp:379] Scheduler::registered took 2.407994ms
I1024 16:00:03.264247 5249 sched.cpp:195] New master at master@127.0.0.1:48588
I1024 16:00:03.264518 5249 sched.cpp:210] Scheduler::disconnected took 18858ns
I1024 16:00:03.264711 5249 sched.cpp:281] Authenticating with master master@127.0.0.1:48588
I1024 16:00:03.265097 5252 authenticatee.hpp:124] Creating new client SASL connection
I1024 16:00:03.265178 5252 master.cpp:1702] Authenticating framework at scheduler(72)@127.0.0.1:48588
I1024 16:00:03.265323 5252 authenticator.hpp:140] Creating new server SASL connection
I1024 16:00:03.265396 5252 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5
I1024 16:00:03.265410 5252 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5'
I1024 16:00:03.265439 5252 authenticator.hpp:243] Received SASL authentication start
I1024 16:00:03.265473 5252 authenticator.hpp:325] Authentication requires more steps
I1024 16:00:03.265499 5252 authenticatee.hpp:258] Received SASL authentication step
I1024 16:00:03.265527 5252 authenticator.hpp:271] Received SASL authentication step
I1024 16:00:03.265542 5252 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1024 16:00:03.265550 5252 auxprop.cpp:153] Looking up auxiliary property '*userPassword'
I1024 16:00:03.265559 5252 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1024 16:00:03.265568 5252 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1024 16:00:03.265574 5252 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1024 16:00:03.265600 5252 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1024 16:00:03.265619 5252 authenticator.hpp:317] Authentication success
I1024 16:00:03.265646 5252 authenticatee.hpp:298] Authentication success
I1024 16:00:03.265676 5252 master.cpp:1742] Successfully authenticated framework at scheduler(72)@127.0.0.1:48588
I1024 16:00:03.265780 5229 master.cpp:557] Master terminating
I1024 16:00:03.265799 5229 master.cpp:209] Shutting down master
I1024 16:00:03.266067 5249 sched.cpp:630] Stopping framework '201310241600-16777343-48588-5229-0000'
tests/fault_tolerance_tests.cpp:615: Failure
Actual function call count doesn't match EXPECT_CALL(sched, registered(&driver, _, _))...
Expected: to be called once
Actual: never called - unsatisfied and active
[ FAILED ] FaultToleranceTest.MasterFailover (15 ms)