Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
This doesn't seem to be a problem on Jenkins. Wondering what was causing the problem on my test box...
Test command:
sudo MESOS_VERBOSE=1 GLOG_v=1 make check -j GTEST_FILTER="CgroupsIsolatorTest/ROOT_CGROUPS_BalloonFramework*"
Log:
[ RUN ] CgroupsIsolatorTest.ROOT_CGROUPS_BalloonFramework_NoBuffer
Using temporary directory '/tmp/CgroupsIsolatorTest_ROOT_CGROUPS_BalloonFramework_NoBuffer_UkYn2q'
Launched master at 56065
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 21:56:49.593399 56065 process.cpp:1555] libprocess is initialized on 127.0.0.1:5432 for 24 cpus
I1014 21:56:49.593868 56065 logging.cpp:110] Logging to STDERR
I1014 21:56:49.594128 56065 main.cpp:114] Build: 2013-10-14 18:18:08 by root
I1014 21:56:49.594156 56065 main.cpp:115] Starting Mesos master
I1014 21:56:49.594457 56084 master.cpp:284] Master started on 127.0.0.1:5432
I1014 21:56:49.594609 56084 master.cpp:299] Master ID: 201310142156-16777343-5432-56065
I1014 21:56:49.594671 56084 master.cpp:302] Master only allowing authenticated frameworks to register!
I1014 21:56:49.595432 56101 hierarchical_allocator_process.hpp:302] Initializing hierarchical allocator process with master : master@127.0.0.1:5432
I1014 21:56:49.595450 56090 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:56:49.596568 56084 master.cpp:697] Elected as master!
I1014 21:56:50.595949 56086 hierarchical_allocator_process.hpp:726] No resources available to allocate!
I1014 21:56:50.596043 56086 hierarchical_allocator_process.hpp:688] Performed allocation for 0 slaves in 108.976us
Launched slave at 56114
I1014 21:56:51.596371 56089 hierarchical_allocator_process.hpp:726] No resources available to allocate!
I1014 21:56:51.596426 56089 hierarchical_allocator_process.hpp:688] Performed allocation for 0 slaves in 66.034us
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 21:56:51.600865 56114 process.cpp:1555] libprocess is initialized on 10.35.255.108:5051 for 24 cpus
I1014 21:56:51.601160 56114 logging.cpp:110] Logging to STDERR
I1014 21:56:51.601503 56114 main.cpp:119] Creating "cgroups" isolator
I1014 21:56:51.601678 56114 main.cpp:127] Build: 2013-10-14 18:18:08 by root
I1014 21:56:51.601706 56114 main.cpp:128] Starting Mesos slave
I1014 21:56:51.603461 56137 slave.cpp:108] Slave started on 1)@10.35.255.108:5051
I1014 21:56:51.604014 56137 slave.cpp:208] Slave resources: cpus
:1; mem:96; disk:454895; ports:[31000-32000]
I1014 21:56:51.605553 56138 cgroups_isolator.cpp:224] Using /tmp/mesos_test_cgroup as cgroups hierarchy root
I1014 21:56:51.606158 56137 slave.cpp:547] New master detected at master@127.0.0.1:5432
I1014 21:56:51.606353 56139 status_update_manager.cpp:157] New master detected at master@127.0.0.1:5432
I1014 21:56:51.606314 56137 slave.cpp:562] Postponing registration until recovery is complete
I1014 21:56:52.597699 56086 hierarchical_allocator_process.hpp:726] No resources available to allocate!
I1014 21:56:52.597754 56086 hierarchical_allocator_process.hpp:688] Performed allocation for 0 slaves in 64.23us
I1014 21:56:52.604190 56138 cgroups_isolator.cpp:817] Recovering isolator
I1014 21:56:52.604665 56138 slave.cpp:399] Finished recovery
I1014 21:56:52.605499 56089 master.cpp:1248] Attempting to register slave on smfd-atr-11-sr1.devel.twitter.com at slave(1)@10.35.255.108:5051
I1014 21:56:52.605533 56089 master.cpp:2502] Adding slave 201310142156-16777343-5432-56065-0 at smfd-atr-11-sr1.devel.twitter.com with cpus:1; mem:96; disk:454895; ports:[31000-32000]
I1014 21:56:52.605934 56080 hierarchical_allocator_process.hpp:445] Added slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) with cpus:1; mem:96; disk:454895; ports:[31000-32000] (and cpus:1; mem:96; disk:454895; ports:[31000-32000] available)
I1014 21:56:52.606008 56080 hierarchical_allocator_process.hpp:708] Performed allocation for slave 201310142156-16777343-5432-56065-0 in 17.439us
I1014 21:56:52.605998 56146 slave.cpp:613] Registered with master master@127.0.0.1:5432; given slave ID 201310142156-16777343-5432-56065-0
Enabling authentication for the framework
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1014 21:56:53.598536 56084 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 25.985us
I1014 21:56:53.598472 56159 process.cpp:1555] libprocess is initialized on 10.35.255.108:45654 for 24 cpus
I1014 21:56:53.599010 56159 logging.cpp:110] Logging to STDERR
I1014 21:56:53.601263 56182 sched.cpp:195] New master at master@127.0.0.1:5432
I1014 21:56:53.601438 56182 sched.cpp:281] Authenticating with master master@127.0.0.1:5432
I1014 21:56:53.601858 56186 authenticatee.hpp:100] Initializing client SASL
I1014 21:56:53.603885 56186 authenticatee.hpp:124] Creating new client SASL connection
I1014 21:56:53.604517 56089 master.cpp:1723] Authenticating framework at scheduler(1)@10.35.255.108:45654
I1014 21:56:53.604909 56102 authenticator.hpp:83] Initializing server SASL
I1014 21:56:53.606436 56102 auxprop.cpp:45] Initialized in-memory auxiliary property plugin
I1014 21:56:53.606469 56102 authenticator.hpp:140] Creating new server SASL connection
I1014 21:56:53.606798 56182 authenticatee.hpp:212] Received SASL authentication mechanisms: CRAM-MD5
I1014 21:56:53.606838 56182 authenticatee.hpp:238] Attempting to authenticate with mechanism 'CRAM-MD5'
I1014 21:56:53.607024 56102 authenticator.hpp:243] Received SASL authentication start
I1014 21:56:53.607133 56102 authenticator.hpp:325] Authentication requires more steps
I1014 21:56:53.607283 56191 authenticatee.hpp:258] Received SASL authentication step
I1014 21:56:53.607574 56096 authenticator.hpp:271] Received SASL authentication step
I1014 21:56:53.607641 56096 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false
I1014 21:56:53.607661 56096 auxprop.cpp:153] Looking up auxiliary property '*userPassword'
I1014 21:56:53.607683 56096 auxprop.cpp:153] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I1014 21:56:53.607699 56096 auxprop.cpp:81] Request to lookup properties for user: 'test-principal' realm: 'smfd-atr-11-sr1.devel.twitter.com' server FQDN: 'smfd-atr-11-sr1.devel.twitter.com' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true
I1014 21:56:53.607712 56096 auxprop.cpp:103] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I1014 21:56:53.607724 56096 auxprop.cpp:103] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I1014 21:56:53.607745 56096 authenticator.hpp:317] Authentication success
I1014 21:56:53.608036 56192 authenticatee.hpp:298] Authentication success
I1014 21:56:53.608242 56196 sched.cpp:326] Successfully authenticated with master master@127.0.0.1:5432
I1014 21:56:53.607980 56093 master.cpp:1763] Successfully authenticated framework at scheduler(1)@10.35.255.108:45654
I1014 21:56:53.608603 56093 master.cpp:768] Received registration request from scheduler(1)@10.35.255.108:45654
I1014 21:56:53.608747 56093 master.cpp:786] Registering framework 201310142156-16777343-5432-56065-0000 at scheduler(1)@10.35.255.108:45654
I1014 21:56:53.609491 56185 sched.cpp:365] Framework registered with 201310142156-16777343-5432-56065-0000
Registered
I1014 21:56:53.609565 56185 sched.cpp:379] Scheduler::registered took 15.017us
I1014 21:56:53.609279 56093 hierarchical_allocator_process.hpp:332] Added framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.609715 56093 hierarchical_allocator_process.hpp:752] Offering cpus:1; mem:96; disk:454895; ports:[31000-32000] on slave 201310142156-16777343-5432-56065-0 to framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.610141 56093 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 468.125us
I1014 21:56:53.610266 56096 master.hpp:389] Adding offer 201310142156-16777343-5432-56065-0 with resources cpus:1; mem:96; disk:454895; ports:[31000-32000] on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
I1014 21:56:53.610985 56096 master.cpp:1689] Sending 1 offers to framework 201310142156-16777343-5432-56065-0000
Resource offers received
Starting the task
I1014 21:56:53.611999 56196 sched.cpp:472] Scheduler::resourceOffers took 99.793us
I1014 21:56:53.612448 56099 master.cpp:2015] Processing reply for offer 201310142156-16777343-5432-56065-0 on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) for framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.612907 56099 master.hpp:361] Adding task 1 with resources mem:32 on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
I1014 21:56:53.613489 56099 master.cpp:2139] Launching task 1 of framework 201310142156-16777343-5432-56065-0000 with resources mem:32 on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
I1014 21:56:53.614058 56092 hierarchical_allocator_process.hpp:547] Framework 201310142156-16777343-5432-56065-0000 left cpus:1; disk:454895; ports:[31000-32000] unused on slave 201310142156-16777343-5432-56065-0
I1014 21:56:53.614356 56139 slave.cpp:786] Got assigned task 1 for framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.614683 56139 slave.cpp:897] Launching task 1 for framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.616145 56139 slave.cpp:1008] Queuing task '1' for executor default of framework '201310142156-16777343-5432-56065-0000
I1014 21:56:53.614069 56099 master.hpp:399] Removing offer 201310142156-16777343-5432-56065-0 with resources cpus:1; mem:96; disk:454895; ports:[31000-32000] on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
I1014 21:56:53.616436 56142 slave.cpp:529] Successfully attached file '/tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed'
I1014 21:56:53.614352 56092 hierarchical_allocator_process.hpp:590] Framework 201310142156-16777343-5432-56065-0000 filtered slave 201310142156-16777343-5432-56065-0 for 5secs
I1014 21:56:53.616778 56148 cgroups_isolator.cpp:516] Launching default (/home/jyx/versions/mesos3/build/src/.libs/balloon-executor) in /tmp/tmp.RrXIQ56113/slaves/201310142156-16777343-5432-56065-0/frameworks/201310142156-16777343-5432-56065-0000/executors/default/runs/1c847474-6b5e-47f9-acf7-a3f0a5f534ed with resources mem:64 for framework 201310142156-16777343-5432-56065-0000 in cgroup mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:53.617432 56148 cgroups_isolator.cpp:708] Changing cgroup controls for executor default of framework 201310142156-16777343-5432-56065-0000 with resources mem:64
I1014 21:56:53.617730 56148 cgroups_isolator.cpp:1089] Updated 'memory.soft_limit_in_bytes' to 64MB for executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.618276 56148 cgroups_isolator.cpp:1124] Updated 'memory.limit_in_bytes' to 64MB for executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.619503 56148 cgroups_isolator.cpp:1177] Started listening for OOM events for executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.620424 56148 cgroups_isolator.cpp:568] Forked executor at = 56198
I1014 21:56:53.649888 56149 slave.cpp:1460] Got registration for executor 'default' of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.650070 56132 cgroups_isolator.cpp:708] Changing cgroup controls for executor default of framework 201310142156-16777343-5432-56065-0000 with resources mem:96
I1014 21:56:53.650190 56149 slave.cpp:1581] Flushing queued task 1 for executor 'default' of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.650724 56132 cgroups_isolator.cpp:1089] Updated 'memory.soft_limit_in_bytes' to 96MB for executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.651286 56132 cgroups_isolator.cpp:1124] Updated 'memory.limit_in_bytes' to 96MB for executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.651778 56132 cgroups_isolator.cpp:1200] Discarded memory threshold (64MB) notifier for executor default of framework 201310142156-16777343-5432-56065-0000 with uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:53.757016 56148 cgroups_isolator.cpp:1226] OOM event detected for executor default of framework 201310142156-16777343-5432-56065-0000 with uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed ; invoking OOM handler
I1014 21:56:53.757143 56148 cgroups_isolator.cpp:1272] OOM handler invoked for executor default of framework 201310142156-16777343-5432-56065-0000 with uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:53.976590 56148 cgroups_isolator.cpp:1311] Memory limit exceeded: Requested: 96MB Maximum Used: 96MB
MEMORY STATISTICS:
cache 4096
rss 100659200
mapped_file 0
pgpgin 24630
pgpgout 54
pgfault 27348
pgmajfault 0
inactive_anon 8200192
active_anon 2347008
inactive_file 4096
active_file 0
unevictable 90095616
hierarchical_memory_limit 100663296
total_cache 4096
total_rss 100659200
total_mapped_file 0
total_pgpgin 24630
total_pgpgout 54
total_pgfault 27348
total_pgmajfault 0
total_inactive_anon 8257536
total_active_anon 2347008
total_inactive_file 4096
total_active_file 0
total_unevictable 90034176
I1014 21:56:53.977507 56148 cgroups_isolator.cpp:672] Killing executor default of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:53.979003 56148 cgroups_isolator.cpp:1200] Discarded memory threshold (96MB) notifier for executor default of framework 201310142156-16777343-5432-56065-0000 with uuid 1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:53.997982 56153 process.cpp:986] Socket closed while receiving
I1014 21:56:54.084249 56130 cgroups.cpp:1193] Trying to freeze cgroup /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:54.085297 56130 cgroups.cpp:1232] Successfully froze cgroup /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed after 1 attempts
I1014 21:56:54.087805 56152 cgroups.cpp:1208] Trying to thaw cgroup /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:54.088615 56152 cgroups.cpp:1318] Successfully thawed /tmp/mesos_test_cgroup/mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:54.090497 56152 cgroups_isolator.cpp:1344] Successfully destroyed cgroup mesos_test/framework_201310142156-16777343-5432-56065-0000_executor_default_tag_1c847474-6b5e-47f9-acf7-a3f0a5f534ed
I1014 21:56:54.091753 56149 slave.cpp:2175] Executor 'default' of framework 201310142156-16777343-5432-56065-0000 has terminated with signal Unknown signal 127
I1014 21:56:54.093579 56149 slave.cpp:1793] Handling status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from @0.0.0.0:0
I1014 21:56:54.094251 56149 slave.cpp:3139] Terminating task 1
I1014 21:56:54.095079 56088 master.cpp:1498] Executor default of framework 201310142156-16777343-5432-56065-0000 on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com) exited with status -1
I1014 21:56:54.094638 56148 cgroups_isolator.cpp:702] Asked to update resources for an unknown/killed executor
I1014 21:56:54.094705 56149 status_update_manager.cpp:300] Received status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.095487 56088 hierarchical_allocator_process.hpp:637] Recovered mem:64 (total allocatable: cpus:1; disk:454895; ports:[31000-32000]; mem:64) on slave 201310142156-16777343-5432-56065-0 from framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.096093 56149 status_update_manager.cpp:471] Creating StatusUpdate stream for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.096649 56149 status_update_manager.cpp:351] Forwarding status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432
Task in state TASK_FAILED
Reason: Memory limit exceeded: Requested: 96MB Maximum Used: 96MB
MEMORY STATISTICS:
cache 4096
rss 100659200
mapped_file 0
pgpgin 24630
pgpgout 54
pgfault 27348
pgmajfault 0
inactive_anon 8200192
active_anon 2347008
inactive_file 4096
active_file 0
unevictable 90095616
hierarchical_memory_limit 100663296
total_cache 4096
total_rss 100659200
total_mapped_file 0
total_pgpgin 24630
total_pgpgout 54
total_pgfault 27348
total_pgmajfault 0
total_inactive_anon 8257536
total_active_anon 2347008
total_inactive_file 4096
total_active_file 0
total_unevictable 90034176
I1014 21:56:54.097167 56149 slave.cpp:1912] Status update manager successfully handled status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.098013 56192 sched.cpp:527] Scheduler::statusUpdate took 57.234us
I1014 21:56:54.098096 56192 sched.cpp:654] Aborting framework '201310142156-16777343-5432-56065-0000'
I1014 21:56:54.098150 56192 sched.cpp:542] Not sending status update acknowledgment message because the driver is aborted!
I1014 21:56:54.097627 56091 master.cpp:1448] Status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051
I1014 21:56:54.098402 56091 master.hpp:379] Removing task 1 with resources mem:32 on slave 201310142156-16777343-5432-56065-0 (smfd-atr-11-sr1.devel.twitter.com)
I1014 21:56:54.098795 56091 master.cpp:1000] scheduler(1)@10.35.255.108:45654 asked to deactivate framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.098858 56086 hierarchical_allocator_process.hpp:637] Recovered mem:32 (total allocatable: cpus:1; disk:454895; ports:[31000-32000]; mem:96) on slave 201310142156-16777343-5432-56065-0 from framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.098934 56091 master.cpp:1014] Deactivating framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.099546 56098 hierarchical_allocator_process.hpp:408] Deactivated framework 201310142156-16777343-5432-56065-0000
I1014 21:56:54.595873 56091 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:56:54.599139 56100 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 19.162us
I1014 21:56:55.600440 56102 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 20.653us
I1014 21:56:56.601618 56085 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 63.838us
I1014 21:56:57.602627 56096 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 24.951us
I1014 21:56:58.603863 56099 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 15.374us
I1014 21:56:58.604903 56182 sched.cpp:337] Ignoring authentication timeout because the driver is aborted!
I1014 21:56:59.597434 56095 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:56:59.605849 56090 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 65.767us
I1014 21:57:00.606931 56092 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 22.189us
I1014 21:57:01.607993 56089 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 23.982us
I1014 21:57:02.609069 56080 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 27.794us
I1014 21:57:03.610221 56087 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 63.708us
W1014 21:57:04.098196 56141 status_update_manager.cpp:454] Resending status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:57:04.099505 56141 status_update_manager.cpp:351] Forwarding status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432
I1014 21:57:04.101228 56190 sched.cpp:502] Ignoring task status update message because the driver is aborted!
W1014 21:57:04.100833 56088 master.cpp:1441] Status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
I1014 21:57:04.598647 56086 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:04.611861 56081 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.172us
I1014 21:57:05.613046 56101 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 14.305us
I1014 21:57:06.614200 56098 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 25.934us
I1014 21:57:07.615419 56081 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 56.7us
I1014 21:57:08.616662 56097 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 83.339us
I1014 21:57:09.600827 56101 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:09.618006 56088 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 51.953us
I1014 21:57:10.619179 56086 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 63.038us
I1014 21:57:11.620412 56080 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 104.269us
I1014 21:57:12.621666 56100 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 18.998us
I1014 21:57:13.622977 56090 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 17.009us
W1014 21:57:14.101284 56149 status_update_manager.cpp:454] Resending status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:57:14.102404 56149 status_update_manager.cpp:351] Forwarding status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432
I1014 21:57:14.103987 56180 sched.cpp:502] Ignoring task status update message because the driver is aborted!
W1014 21:57:14.103621 56089 master.cpp:1441] Status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
I1014 21:57:14.602560 56085 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:14.624765 56099 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 20.175us
I1014 21:57:15.625892 56093 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 31.194us
I1014 21:57:16.626806 56082 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 22.441us
I1014 21:57:17.627913 56091 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.554us
I1014 21:57:18.629115 56101 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.692us
I1014 21:57:19.604230 56081 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:19.630450 56103 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 88.214us
I1014 21:57:20.631455 56094 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 14.402us
I1014 21:57:21.632642 56097 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.937us
I1014 21:57:22.634487 56085 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 66.372us
I1014 21:57:23.635640 56080 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 21.919us
W1014 21:57:24.103406 56139 status_update_manager.cpp:454] Resending status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:57:24.104260 56139 status_update_manager.cpp:351] Forwarding status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432
I1014 21:57:24.106067 56182 sched.cpp:502] Ignoring task status update message because the driver is aborted!
W1014 21:57:24.105670 56103 master.cpp:1441] Status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
I1014 21:57:24.605545 56082 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:24.636827 56091 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 15.822us
I1014 21:57:25.637984 56097 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.822us
I1014 21:57:26.639128 56094 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 15.062us
I1014 21:57:27.640239 56101 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.755us
I1014 21:57:28.641381 56089 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 18.619us
I1014 21:57:29.606472 56099 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:29.642806 56100 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 59.527us
I1014 21:57:30.644026 56083 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 21.656us
I1014 21:57:31.645145 56092 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.892us
I1014 21:57:32.646399 56082 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 12.527us
I1014 21:57:33.647538 56080 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 14.082us
W1014 21:57:34.105988 56140 status_update_manager.cpp:454] Resending status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000
I1014 21:57:34.106892 56140 status_update_manager.cpp:351] Forwarding status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 to master@127.0.0.1:5432
I1014 21:57:34.108701 56190 sched.cpp:502] Ignoring task status update message because the driver is aborted!
W1014 21:57:34.108274 56102 master.cpp:1441] Status update TASK_FAILED (UUID: d2166e50-d765-4530-b6d0-b91425e02481) for task 1 of framework 201310142156-16777343-5432-56065-0000 from slave(1)@10.35.255.108:5051 (smfd-atr-11-sr1.devel.twitter.com): error, couldn't lookup task
I1014 21:57:34.608364 56084 master.cpp:85] No whitelist given. Advertising offers for all slaves
I1014 21:57:34.648485 56085 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 14.287us
I1014 21:57:35.649591 56091 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 13.624us
I1014 21:57:36.650728 56098 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 14.247us
I1014 21:57:37.652891 56094 hierarchical_allocator_process.hpp:688] Performed allocation for 1 slaves in 51.175us
...