[YARN-10192] CapacityScheduler stuck in loop rejecting allocation proposals

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.10.0
• Assignee: Unassigned
• Reporter: Jonathan Hung
• Created: 11/Mar/20 17:40
• Updated: 18/Mar/20 18:24

Description

On a 2.10.0 cluster, we observed containers were being scheduled very slowly. Based on logs, it seems to reject a bunch of allocation proposals, then accept a bunch of reserved containers, but very few containers are actually getting allocated:

2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,968 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,977 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,981 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,982 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,985 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,988 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,988 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,989 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,989 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,995 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,074 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8957.grid.linkedin.com:8041
2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,084 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,095 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15460 resource=<memory:2048, vCores:1> queue=misc_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: assignedContainer application attempt=appattempt_1582403122262_15460_000001 container=null queue=misc_default clusterResource=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=misc usedCapacity=0.0031771248 absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.30113637 absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,104 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Failed to accept allocation proposal
2020-03-10 06:31:49,132 INFO com.linkedin.hadoop.yarn.ams.NodeLabelRoutingAMSProcessor: Applying node label cpu to container request from application_1582403122262_15460
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000283 Container Transitioned from ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest  IP=10.150.241.150       OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1582403122262_15460   CONTAINERID=container_e43_1582403122262_15460_01_000283 RESOURCE=<memory:2048, vCores:1>        QUEUENAME=misc_default  NODELABEL=cpu
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000284 Container Transitioned from ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest  IP=10.150.241.150       OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1582403122262_15460   CONTAINERID=container_e43_1582403122262_15460_01_000284 RESOURCE=<memory:2048, vCores:1>        QUEUENAME=misc_default  NODELABEL=cpu
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_e43_1582403122262_15460_01_000285 Container Transitioned from ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest  IP=10.150.241.150       OPERATION=AM Released Container TARGET=SchedulerApp     RESULT=SUCCESS  APPID=application_1582403122262_15460   CONTAINERID=container_e43_1582403122262_15460_01_000285 RESOURCE=<memory:2048, vCores:1>        QUEUENAME=misc_default  NODELABEL=cpu
2020-03-10 06:31:49,132 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate of application :application_1582403122262_15460
2020-03-10 06:31:49,197 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8950.grid.linkedin.com:8041
2020-03-10 06:31:49,198 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,198 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8650.grid.linkedin.com:8041
2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,210 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8696.grid.linkedin.com:8041
2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,219 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8689.grid.linkedin.com:8041
2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,290 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9327.grid.linkedin.com:8041
2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,315 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8617.grid.linkedin.com:8041
2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,320 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9342.grid.linkedin.com:8041
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8941.grid.linkedin.com:8041
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,327 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3167.grid.linkedin.com:8041
2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,335 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,357 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8679.grid.linkedin.com:8041
2020-03-10 06:31:49,358 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,358 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3161.grid.linkedin.com:8041
2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,360 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3197.grid.linkedin.com:8041
2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,363 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9364.grid.linkedin.com:8041
2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,375 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9367.grid.linkedin.com:8041
2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,376 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8668.grid.linkedin.com:8041
2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,380 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,382 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8952.grid.linkedin.com:8041
2020-03-10 06:31:49,383 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,383 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9329.grid.linkedin.com:8041
2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,394 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8682.grid.linkedin.com:8041
2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,399 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,403 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8708.grid.linkedin.com:8041
2020-03-10 06:31:49,404 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,404 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8932.grid.linkedin.com:8041
2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,410 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9340.grid.linkedin.com:8041
2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,415 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9378.grid.linkedin.com:8041
2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,422 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3171.grid.linkedin.com:8041
2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl9365.grid.linkedin.com:8041
2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,430 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8919.grid.linkedin.com:8041
2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,431 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8644.grid.linkedin.com:8041
2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,443 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8614.grid.linkedin.com:8041
2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl3176.grid.linkedin.com:8041
2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,444 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted
2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Trying to fulfill reservation for application application_1582403122262_15459 on node: ltx1-hcl8615.grid.linkedin.com:8041
2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: Reserved container  application=application_1582403122262_15459 resource=<memory:72704, vCores:2> queue=relevance_default cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,445 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Allocation proposal accepted