[MESOS-8065] Implicitly declining offers in ACCEPT call as the launch operation specified no tasks - for GPU based job

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.4.0
• Component/s: docker, framework, gpu
• Labels: docker gpu newbie

Environment

Ubuntu 14.04

• Assignee: Unassigned
• Reporter: Ambuj Pandey
• Created: 10/Oct/17 09:36
• Updated: 13/Nov/17 20:25
• Resolved: 13/Nov/17 20:25

Description

I tried changing the "docker-no-executor-framework" to require GPU resource and use mesos containerizer with a docker image of nvidia/cuda.

This worked for me when i ran this framework with the master IP of one installed with DC/OS. But when I manually compiled Mesos (1.4/1.5) and started agent/master on another machine I face this issue : "Implicitly declining offers in ACCEPT call as the launch operation specified no tasks."
I've tested the nvidia/cuda image to work on that target machine using nvidia-docker.

Configuration:
Master -
./bin/mesos-master.sh pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --filter_gpu_resources=false

Agent (on the same node) -
./bin/mesos-agent.sh --master=pas-lab-server6:5050 --work_dir=/home/sayantan/mesos_temp --isolation="filesystem/linux,cgroups/devices,gpu/nvidia"

I1010 14:48:44.106006 20372 sched.cpp:336] New master detected at master@127.0.1.1:5050
I1010 14:48:44.107053 20372 sched.cpp:351] No credentials provided. Attempting to register without authentication
I1010 14:48:44.113246 20365 sched.cpp:747] Framework registered with 00afc79a-cb51-4f9d-8c78-6000253ae396-0000
Registered!
.Starting task 0 on pas-lab-server6
Hello
Task 0 is in state 3
. . . . . .

The log on master says:

I1010 14:48:44.420087 19698 master.cpp:7766] Sending 1 offers to framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
W1010 14:48:44.422901 19701 master.cpp:3911] Implicitly declining offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1 ] in ACCEPT call for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 as the launch operation specified no tasks
I1010 14:48:44.423539 19701 master.cpp:9389] Removing offer 00afc79a-cb51-4f9d-8c78-6000253ae396-O1
I1010 14:48:44.423607 19701 master.cpp:4196] Processing ACCEPT call for offers: [ 00afc79a-cb51-4f9d-8c78-6000253ae396-O1 ] on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 at slave(1)@127.0.1.1:5051 (pas-lab-server6) for framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 (Docker No Executor Framework (C++)) at scheduler-2beb12b1-971d-4f05-b023-498665053465@127.0.1.1:45628
I1010 14:48:44.425109 19691 hierarchical.cpp:887] Updated allocation of framework 00afc79a-cb51-4f9d-8c78-6000253ae396-0000 on agent 00afc79a-cb51-4f9d-8c78-6000253ae396-S0 from gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000] to gpus(allocated: *):2; cpus(allocated: *):40; mem(allocated: *):63421; disk(allocated: *):868675; ports(allocated: *):[31000-32000]