[MESOS-436] FaultToleranceTest.SchedulerFailover test is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Vinod Kone
• Reporter: Vinod Kone
• Created: 17/Apr/13 01:43
• Updated: 30/May/13 23:42
• Resolved: 30/May/13 23:42

Description

[ RUN ] FaultToleranceTest.SchedulerFailover
I0416 15:06:26.879618 8632 master.cpp:309] Master started on 67.195.138.60:58457
I0416 15:06:26.879665 8632 master.cpp:324] Master ID: 201304161506-1015726915-58457-8569
I0416 15:06:26.880308 8627 hierarchical_allocator_process.hpp:236] Initializing hierarchical allocator process with master : master@67.195.138.60:58457
I0416 15:06:26.880391 8634 sched.cpp:182] New master at master@67.195.138.60:58457
W0416 15:06:26.880410 8626 master.cpp:81] No whitelist given. Advertising offers for all slaves
I0416 15:06:26.880056 8633 slave.cpp:203] Slave started on 53)@67.195.138.60:58457
I0416 15:06:26.880465 8632 master.cpp:603] Elected as master!
I0416 15:06:26.883709 8632 master.cpp:646] Registering framework 201304161506-1015726915-58457-8569-0000 at scheduler(41)@67.195.138.60:58457
I0416 15:06:26.884176 8628 sched.cpp:217] Framework registered with 201304161506-1015726915-58457-8569-0000
I0416 15:06:26.884181 8632 hierarchical_allocator_process.hpp:268] Added framework 201304161506-1015726915-58457-8569-0000
I0416 15:06:26.885404 8632 hierarchical_allocator_process.hpp:668] No resources available to allocate!
I0416 15:06:26.885946 8632 hierarchical_allocator_process.hpp:599] Performed allocation for 0 slaves in 542.782000000000039us
I0416 15:06:26.883235 8633 slave.cpp:204] Slave resources: cpus=2; mem=1024; ports=[31000-32000]; disk=1024
I0416 15:06:26.887214 8634 process_isolator.cpp:300] Recovering isolator
I0416 15:06:26.887229 8633 slave.cpp:453] New master detected at master@67.195.138.60:58457
I0416 15:06:26.884953 8627 sched.cpp:182] New master at master@67.195.138.60:58457
I0416 15:06:26.889400 8626 master.cpp:681] Re-registering framework 201304161506-1015726915-58457-8569-0000 at scheduler(42)@67.195.138.60:58457
I0416 15:06:26.889853 8626 master.cpp:700] Framework 201304161506-1015726915-58457-8569-0000 failed over
I0416 15:06:26.888891 8633 slave.cpp:377] Finished recovery
I0416 15:06:26.888903 8634 status_update_manager.cpp:132] New master detected at master@67.195.138.60:58457
I0416 15:06:26.890351 8626 sched.cpp:217] Framework registered with 201304161506-1015726915-58457-8569-0000
tests/fault_tolerance_tests.cpp:263: Failure
Value of: driver1.stop()
Actual: 4
Expected: DRIVER_ABORTED
Which is: 3
I0416 15:06:26.890355 8627 sched.cpp:413] Got error 'Framework failed over'
I0416 15:06:26.892359 8627 sched.cpp:422] Stopping framework '201304161506-1015726915-58457-8569-0000'
I0416 15:06:26.891923 8626 sched.cpp:422] Stopping framework '201304161506-1015726915-58457-8569-0000'
I0416 15:06:26.890974 8630 master.cpp:968] Attempting to register slave on janus.apache.org at slave(53)@67.195.138.60:58457
I0416 15:06:26.894078 8630 master.cpp:1224] Master now considering a slave at janus.apache.org:58457 as active
I0416 15:06:26.894557 8630 master.cpp:1862] Adding slave 201304161506-1015726915-58457-8569-0 at janus.apache.org with cpus=2; mem=1024; ports=[31000-32000]; disk=1024
I0416 15:06:26.895086 8633 slave.cpp:487] Registered with master; given slave ID 201304161506-1015726915-58457-8569-0
I0416 15:06:26.895692 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304122015-1015726915-48096-7061-0' for removal
I0416 15:06:26.896188 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304160210-1015726915-50493-24062-1' for removal
I0416 15:06:26.895133 8630 master.cpp:477] Master terminating
I0416 15:06:26.895159 8634 hierarchical_allocator_process.hpp:395] Added slave 201304161506-1015726915-58457-8569-0 (janus.apache.org) with cpus=2; mem=1024; ports=[31000-32000]; disk=1024 (and cpus=2; mem=1024; ports=[31000-32000]; disk=1024 available)
I0416 15:06:26.897363 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304150351-1015726915-54612-31975-2' for removal
I0416 15:06:26.897824 8630 slave.cpp:424] Slave asked to shut down by master@67.195.138.60:58457
I0416 15:06:26.899477 8630 slave.cpp:386] Slave terminating
I0416 15:06:26.898480 8634 hierarchical_allocator_process.hpp:660] Found available resources: cpus=2; mem=1024; ports=[31000-32000]; disk=1024 on slave 201304161506-1015726915-58457-8569-0
I0416 15:06:26.900465 8634 hierarchical_allocator_process.hpp:686] Offering cpus=2; mem=1024; ports=[31000-32000]; disk=1024 on slave 201304161506-1015726915-58457-8569-0 to framework 201304161506-1015726915-58457-8569-0000
I0416 15:06:26.897832 8569 master.cpp:283] Shutting down master
I0416 15:06:26.898938 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304040412-1015726915-46073-7047-1' for removal
I0416 15:06:26.902230 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304122029-1015726915-59546-7109-1' for removal
I0416 15:06:26.902729 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304160201-1015726915-38500-17544-0' for removal
I0416 15:06:26.903275 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304122054-1015726915-54227-21044-0' for removal
I0416 15:06:26.901274 8634 hierarchical_allocator_process.hpp:619] Performed allocation for slave 201304161506-1015726915-58457-8569-0 in 2.803825000000000ms
I0416 15:06:26.905027 8627 gc.cpp:56] Scheduling '/tmp/mesos/0/slaves/201304122054-1015726915-33795-21534-0' for removal
[ FAILED ] FaultToleranceTest.SchedulerFailover (28 ms)