[MESOS-792] FaultToleranceTest.SchedulerFailoverFrameworkMessage is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15.0

Environment

fedora 18

• Assignee: Vinod Kone
• Reporter: Vinod Kone
• Created: 01/Nov/13 19:34
• Updated: 02/Nov/13 00:30
• Resolved: 02/Nov/13 00:30

Description

Observed this on CI.

	
[ RUN      ] FaultToleranceTest.SchedulerFailoverFrameworkMessage
I1031 15:37:26.325614 30608 master.cpp:293] Master started on 127.0.0.1:53353
I1031 15:37:26.325660 30608 master.cpp:308] Master ID: 201310311537-16777343-53353-30587
I1031 15:37:26.325669 30608 master.cpp:311] Master only allowing authenticated frameworks to register!
I1031 15:37:26.325999 30608 master.cpp:706] Elected as master!
I1031 15:37:26.326123 30608 hierarchical_allocator_process.hpp:302] Initializing hierarchical allocator process with master : master@127.0.0.1:53353
I1031 15:37:26.327484 30609 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1031 15:37:26.327553 30609 slave.cpp:109] Slave started on 39)@127.0.0.1:53353
I1031 15:37:26.327635 30609 slave.cpp:209] Slave resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1031 15:37:26.328011 30609 slave.cpp:481] New master detected at master@127.0.0.1:53353
I1031 15:37:26.328037 30609 slave.cpp:496] Postponing registration until recovery is complete
I1031 15:37:26.328111 30609 sched.cpp:195] New master at master@127.0.0.1:53353
I1031 15:37:26.328130 30609 sched.cpp:281] Authenticating with master master@127.0.0.1:53353
I1031 15:37:26.328969 30608 status_update_manager.cpp:158] New master detected at master@127.0.0.1:53353
I1031 15:37:26.329762 30608 state.cpp:33] Recovering state from '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_lThYiC/meta'
I1031 15:37:26.329918 30608 status_update_manager.cpp:180] Recovering status update manager
I1031 15:37:26.330097 30608 slave.cpp:2737] Finished recovery
I1031 15:37:26.330960 30609 authenticatee.hpp:124] Creating new client SASL connection
I1031 15:37:26.331032 30609 master.cpp:1695] Authenticating framework at scheduler(55)@127.0.0.1:53353
I1031 15:37:26.331147 30609 authenticator.hpp:140] Creating new server SASL connection
I1031 15:37:26.331867 30608 master.cpp:1232] Attempting to register slave on localhost.localdomain at slave(39)@127.0.0.1:53353
I1031 15:37:26.331890 30608 master.cpp:2474] Adding slave 201310311537-16777343-53353-30587-0 at localhost.localdomain with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I1031 15:37:26.331982 30608 slave.cpp:547] Registered with master master@127.0.0.1:53353; given slave ID 201310311537-16777343-53353-30587-0
I1031 15:37:26.332041 30608 hierarchical_allocator_process.hpp:445] Added slave 201310311537-16777343-53353-30587-0 (localhost.localdomain) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (and cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] available)
I1031 15:37:26.332078 30608 hierarchical_allocator_process.hpp:708] Performed allocation for slave 201310311537-16777343-53353-30587-0 in 7336ns
I1031 15:37:26.332772 30609 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5
I1031 15:37:26.332793 30609 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5'
I1031 15:37:26.332813 30609 authenticator.hpp:243] Received SASL authentication start
I1031 15:37:26.332856 30609 authenticator.hpp:325] Authentication requires more steps
I1031 15:37:26.332876 30609 authenticatee.hpp:258] Received SASL authentication step
I1031 15:37:26.332903 30609 authenticator.hpp:271] Received SASL authentication step
I1031 15:37:26.332918 30609 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
I1031 15:37:26.332926 30609 auxprop.cpp:153] Looking up auxiliary property '*userPassword'
I1031 15:37:26.332937 30609 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1031 15:37:26.332947 30609 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
I1031 15:37:26.332953 30609 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1031 15:37:26.332959 30609 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1031 15:37:26.332969 30609 authenticator.hpp:317] Authentication success
I1031 15:37:26.332993 30609 authenticatee.hpp:298] Authentication success
I1031 15:37:26.333030 30609 master.cpp:1735] Successfully authenticated framework at scheduler(55)@127.0.0.1:53353
I1031 15:37:26.333091 30609 sched.cpp:326] Successfully authenticated with master master@127.0.0.1:53353
I1031 15:37:26.333892 30609 master.cpp:764] Received registration request from scheduler(55)@127.0.0.1:53353
I1031 15:37:26.335126 30609 master.cpp:782] Registering framework 201310311537-16777343-53353-30587-0000 at scheduler(55)@127.0.0.1:53353
I1031 15:37:26.335855 30609 sched.cpp:365] Framework registered with 201310311537-16777343-53353-30587-0000
I1031 15:37:26.335883 30609 sched.cpp:379] Scheduler::registered took 12158ns
I1031 15:37:26.335906 30609 hierarchical_allocator_process.hpp:332] Added framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.335940 30609 hierarchical_allocator_process.hpp:752] Offering cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201310311537-16777343-53353-30587-0 to framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.336724 30609 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 803841ns
I1031 15:37:26.337607 30608 master.hpp:419] Adding offer 201310311537-16777343-53353-30587-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201310311537-16777343-53353-30587-0 (localhost.localdomain)
I1031 15:37:26.338348 30608 master.cpp:1661] Sending 1 offers to framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.339200 30608 sched.cpp:472] Scheduler::resourceOffers took 56036ns
I1031 15:37:26.340457 30607 master.cpp:1987] Processing reply for offer 201310311537-16777343-53353-30587-0 on slave 201310311537-16777343-53353-30587-0 (localhost.localdomain) for framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.340589 30607 master.hpp:391] Adding task 1 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201310311537-16777343-53353-30587-0 (localhost.localdomain)
I1031 15:37:26.340636 30607 master.cpp:2111] Launching task 1 of framework 201310311537-16777343-53353-30587-0000 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201310311537-16777343-53353-30587-0 (localhost.localdomain)
I1031 15:37:26.340736 30607 master.hpp:429] Removing offer 201310311537-16777343-53353-30587-0 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on slave 201310311537-16777343-53353-30587-0 (localhost.localdomain)
I1031 15:37:26.340806 30607 slave.cpp:721] Got assigned task 1 for framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.340993 30607 slave.cpp:832] Launching task 1 for framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.344346 30606 exec.cpp:174] Executor started at: executor(14)@127.0.0.1:53353 with pid 30587
I1031 15:37:26.345198 30607 slave.cpp:942] Queuing task '1' for executor default of framework '201310311537-16777343-53353-30587-0000
I1031 15:37:26.346392 30607 slave.cpp:463] Successfully attached file '/tmp/FaultToleranceTest_SchedulerFailoverFrameworkMessage_lThYiC/slaves/201310311537-16777343-53353-30587-0/frameworks/201310311537-16777343-53353-30587-0000/executors/default/runs/bb25870a-943a-4cfd-a74c-2768693dcf8e'
I1031 15:37:26.347586 30607 slave.cpp:1404] Got registration for executor 'default' of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.348947 30608 exec.cpp:198] Executor registered on slave 201310311537-16777343-53353-30587-0
I1031 15:37:26.349812 30608 exec.cpp:210] Executor::registered took 17746ns
I1031 15:37:26.350829 30607 slave.cpp:1525] Flushing queued task 1 for executor 'default' of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.352150 30609 exec.cpp:285] Executor asked to run task '1'
I1031 15:37:26.352915 30609 exec.cpp:294] Executor::launchTask took 716050ns
I1031 15:37:26.353682 30609 exec.cpp:505] Executor sending status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.354531 30607 slave.cpp:2067] Monitoring executor default of framework 201310311537-16777343-53353-30587-0000 forked at pid 30587
I1031 15:37:26.355893 30607 slave.cpp:1738] Handling status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000 from executor(14)@127.0.0.1:53353
I1031 15:37:26.357569 30607 status_update_manager.cpp:305] Received status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.357594 30607 status_update_manager.cpp:476] Creating StatusUpdate stream for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.357645 30607 status_update_manager.cpp:356] Forwarding status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000 to master@127.0.0.1:53353
I1031 15:37:26.357772 30607 master.cpp:1418] Status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000 from slave(39)@127.0.0.1:53353
I1031 15:37:26.357823 30607 slave.cpp:1857] Status update manager successfully handled status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.357836 30607 slave.cpp:1863] Sending acknowledgement for status update TASK_RUNNING (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000 to executor(14)@127.0.0.1:53353
I1031 15:37:26.357900 30607 sched.cpp:527] Scheduler::statusUpdate took 18234ns
I1031 15:37:26.357954 30607 exec.cpp:331] Executor received status update acknowledgement a708754a-2f24-403c-a194-d65fe5e452dc for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.358813 30608 status_update_manager.cpp:380] Received status update acknowledgement (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.359752 30608 slave.cpp:1344] Status update manager successfully handled status update acknowledgement (UUID: a708754a-2f24-403c-a194-d65fe5e452dc) for task 1 of framework 201310311537-16777343-53353-30587-0000
I1031 15:37:26.360030 30606 sched.cpp:195] New master at master@127.0.0.1:53353
I1031 15:37:26.360050 30606 sched.cpp:281] Authenticating with master master@127.0.0.1:53353
I1031 15:37:26.360142 30606 authenticatee.hpp:124] Creating new client SASL connection
I1031 15:37:26.360251 30606 master.cpp:1695] Authenticating framework at scheduler(56)@127.0.0.1:53353
I1031 15:37:26.360379 30606 authenticator.hpp:140] Creating new server SASL connection
I1031 15:37:26.360450 30606 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5
I1031 15:37:26.360466 30606 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5'
I1031 15:37:26.360488 30606 authenticator.hpp:243] Received SASL authentication start
I1031 15:37:26.360525 30606 authenticator.hpp:325] Authentication requires more steps
I1031 15:37:26.360545 30606 authenticatee.hpp:258] Received SASL authentication step
I1031 15:37:26.360573 30606 authenticator.hpp:271] Received SASL authentication step
I1031 15:37:26.360589 30606 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
I1031 15:37:26.360596 30606 auxprop.cpp:153] Looking up auxiliary property '*userPassword'
I1031 15:37:26.360609 30606 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1031 15:37:26.360617 30606 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'localhost.localdomain' server FQDN: 'localhost.localdomain' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
I1031 15:37:26.360625 30606 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1031 15:37:26.360630 30606 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1031 15:37:26.360641 30606 authenticator.hpp:317] Authentication success
I1031 15:37:26.360661 30606 authenticatee.hpp:298] Authentication success
I1031 15:37:26.360697 30606 master.cpp:1735] Successfully authenticated framework at scheduler(56)@127.0.0.1:53353
I1031 15:37:26.360755 30606 sched.cpp:326] Successfully authenticated with master master@127.0.0.1:53353
I1031 15:37:26.360833 30606 master.cpp:852] Re-registering framework 201310311537-16777343-53353-30587-0000 at scheduler(56)@127.0.0.1:53353
I1031 15:37:26.360852 30606 master.cpp:878] Framework 201310311537-16777343-53353-30587-0000 failed over
I1031 15:37:26.361002 30608 sched.cpp:614] Got error 'Framework failed over'
I1031 15:37:26.361035 30608 sched.cpp:625] Scheduler::error took 11128ns
I1031 15:37:26.361047 30608 sched.cpp:654] Aborting framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:26.361070 30608 sched.cpp:365] Framework registered with 201310311537-16777343-53353-30587-0000
I1031 15:37:26.361093 30608 sched.cpp:379] Scheduler::registered took 12636ns
I1031 15:37:26.361246 30608 slave.cpp:1916] Sending message for framework 201310311537-16777343-53353-30587-0000 to scheduler(55)@127.0.0.1:53353
I1031 15:37:26.361290 30608 sched.cpp:591] Ignoring framework message because the driver is aborted!
W1031 15:37:26.361332 30606 master.cpp:987] scheduler(55)@127.0.0.1:53353 tried to deactivate framework; expecting scheduler(56)@127.0.0.1:53353
I1031 15:37:26.361352 30606 slave.cpp:1290] Updating framework 201310311537-16777343-53353-30587-0000 pid to scheduler(56)@127.0.0.1:53353
I1031 15:37:26.493057 31090 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:26.493365 31097 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:26.493577 31098 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:27.233518 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:27.300107 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:27.327317 30606 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 43163ns
I1031 15:37:27.356959 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:28.234324 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:28.300559 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:28.327744 30606 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 43002ns
I1031 15:37:28.358132 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:29.235252 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:29.301694 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:29.328815 30608 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 42776ns
I1031 15:37:29.359335 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:30.236479 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:30.303951 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:30.329061 30608 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 62620ns
I1031 15:37:30.360592 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:30.572002 31215 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:30.572233 31208 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:30.578310 31204 exec.cpp:84] Committing suicide by killing the process group
I1031 15:37:31.237666 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:31.305024 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:31.328485 30608 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1031 15:37:31.329763 30608 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 46071ns
I1031 15:37:31.331987 30608 sched.cpp:337] Ignoring authentication timeout because the driver is aborted!
I1031 15:37:31.361563 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:32.238627 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:32.306056 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:32.330296 30609 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 54679ns
I1031 15:37:32.362678 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:33.239739 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:33.309238 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:33.331384 30609 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 55578ns
I1031 15:37:33.363818 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:34.240943 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:34.310294 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:34.332397 30607 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 51003ns
I1031 15:37:34.364899 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:35.242014 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:35.311414 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:35.333597 30607 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 58980ns
I1031 15:37:35.366199 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:36.243355 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:36.312538 30607 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:36.328618 30606 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1031 15:37:36.333750 30606 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 43090ns
tests/fault_tolerance_tests.cpp:1356: Failure
Failed to wait 10secs for frameworkMessage
I1031 15:37:36.367251 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:37.204871 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:37.244323 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:37.315712 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:37.335002 30606 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 51048ns
I1031 15:37:37.368432 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:38.205508 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:38.244714 30606 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:38.317124 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:38.335305 30608 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 53266ns
I1031 15:37:38.369844 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:39.205935 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:39.247402 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:39.317607 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:39.335857 30608 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 45013ns
I1031 15:37:39.371294 30608 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:40.207412 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:40.248796 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:40.318120 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:40.336407 30609 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 52789ns
I1031 15:37:40.374876 30609 monitor.cpp:187] Publishing resource usage for executor 'default' of framework '201310311537-16777343-53353-30587-0000'
I1031 15:37:41.208813 30609 monitor.cpp:187] Publishing re