Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-6054

Agent Crash with Malformed UUID when doing TaskUpdate

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: scheduler api
    • Labels:
      None
    • Environment:

      Ubuntu 14.04, Mesos 1.0.0-2.0.89.ubuntu1404, Marathon 1.1.2

      Description

      When using the HTTP API using protobufs, if the UUID in a TaskUpdate is malformed (in this case, was using a UUID that was base64 encoded), it would cause the Agent where the executor is running on to crash and restart.

      Here is a JSON dump of the protobuf used:

      {
        "executor_id": {
          "value": "executor-scaleio1"
        },
        "framework_id": {
          "value": "ac8545a7-f8fc-431e-bc36-0239c4460658-0002"
        },
        "type": 2,
        "update": {
          "status": {
            "task_id": {
              "value": "scaleio1"
            },
            "state": 1,
            "source": 2,
            "executor_id": {
              "value": "executor-scaleio1"
            },
            "uuid": "WVdVd01EQTFNakF0TkdVeU9TMDBNell3TFdJMk4yUXRPR05sT1RFNU56VmlPREUw"
          }
        }
      }
      

      In the master it looks like is processes the accept calls… but after it processes all of them, it looks like the agents are immediately being disconnected:

      ...
      ...
      I0816 17:53:09.974340  4010 master.cpp:3342] Processing ACCEPT call for offers: [ 2bf179c3-004a-49e3-98ab-5a75fa773522-O80 ] on agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com) for framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (ScaleIO Framework)
      W0816 17:53:09.974578  4010 validation.cpp:647] Executor executor-scaleio4 for task scaleio4 uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
      W0816 17:53:09.974604  4010 validation.cpp:659] Executor executor-scaleio4 for task scaleio4 uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
      I0816 17:53:09.974645  4010 master.cpp:7439] Adding task scaleio4 with resources cpus(*):1; mem(*):2048 on agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:09.974668  4010 master.cpp:3831] Launching task scaleio4 of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (ScaleIO Framework) with resources cpus(*):1; mem(*):2048 on agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.306182  4010 master.cpp:1245] Agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com) disconnected
      I0816 17:53:11.306335  4010 master.cpp:2784] Disconnecting agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.306520  4010 master.cpp:2803] Deactivating agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.306676  4010 master.cpp:1264] Removing framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (ScaleIO Framework) from disconnected agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com) because the framework is not checkpointing
      I0816 17:53:11.306798  4010 master.cpp:6448] Removing framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (ScaleIO Framework) from agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.306882  4010 master.cpp:6833] Updating the state of task scaleio4 of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (latest state: TASK_LOST, status update state: TASK_LOST)
      I0816 17:53:11.306778  4013 hierarchical.cpp:571] Agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 deactivated
      I0816 17:53:11.307140  4010 master.cpp:6899] Removing task scaleio4 with resources cpus(*):1; mem(*):2048 of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 on agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.307312  4010 master.cpp:5190] Sending status update TASK_LOST for task scaleio4 of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 'Slave ec2-52-89-227-184.us-west-2.compute.amazonaws.com disconnected'
      I0816 17:53:11.307533  4010 master.cpp:6928] Removing executor 'executor-scaleio4' with resources  of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 on agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S7 at slave(1)@172.31.22.211:5051 (ec2-52-89-227-184.us-west-2.compute.amazonaws.com)
      I0816 17:53:11.472939  4017 master.cpp:1245] Agent 2bf179c3-004a-49e3-98ab-5a75fa773522-S4 at slave(1)@172.31.17.252:5051 (ec2-52-88-195-213.us-west-2.compute.amazonaws.com) disconnected
      ...
      ...
      

      The agent receives the POST from the executor:

      ...
      ...
      I0816 17:51:09.001888  1237 slave.cpp:4591] Current disk usage 31.86%. Max allowed age: 4.069593432939398days
      I0816 17:52:09.002300  1236 slave.cpp:4591] Current disk usage 31.86%. Max allowed age: 4.069545128332523days
      I0816 17:53:09.002799  1234 slave.cpp:4591] Current disk usage 31.86%. Max allowed age: 4.069496823725636days
      I0816 17:53:10.033020  1240 slave.cpp:1495] Got assigned task scaleio3 for framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001
      I0816 17:53:10.033210  1240 slave.cpp:1614] Launching task scaleio3 for framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001
      I0816 17:53:10.033980  1240 paths.cpp:528] Trying to chown '/tmp/mesos/slaves/2bf179c3-004a-49e3-98ab-5a75fa773522-S5/frameworks/2bf179c3-004a-49e3-98ab-5a75fa773522-0001/executors/executor-scaleio3/runs/9aa4ee18-350b-4a65-a36b-eef9449f5d11' to user 'root'
      I0816 17:53:10.036744  1240 slave.cpp:5674] Launching executor executor-scaleio3 of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 with resources  in work directory '/tmp/mesos/slaves/2bf179c3-004a-49e3-98ab-5a75fa773522-S5/frameworks/2bf179c3-004a-49e3-98ab-5a75fa773522-0001/executors/executor-scaleio3/runs/9aa4ee18-350b-4a65-a36b-eef9449f5d11'
      I0816 17:53:10.036864  1240 slave.cpp:1840] Queuing task 'scaleio3' for executor 'executor-scaleio3' of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001
      I0816 17:53:10.036898  1237 containerizer.cpp:781] Starting container '9aa4ee18-350b-4a65-a36b-eef9449f5d11' for executor 'executor-scaleio3' of framework '2bf179c3-004a-49e3-98ab-5a75fa773522-0001'
      I0816 17:53:10.037387  1240 linux_launcher.cpp:281] Cloning child process with flags = 
      I0816 17:53:10.457927  1234 http.cpp:270] HTTP POST for /slave(1)/api/v1/executor from 172.31.23.107:49326 with User-Agent='scaleio/0.1'
      I0816 17:53:10.458055  1234 slave.cpp:2661] Received Subscribe request for HTTP executor 'executor-scaleio3' of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001
      I0816 17:53:10.462604  1234 slave.cpp:2005] Sending queued task 'scaleio3' to executor 'executor-scaleio3' of framework 2bf179c3-004a-49e3-98ab-5a75fa773522-0001 (via HTTP)
      I0816 17:53:11.464956  1233 http.cpp:270] HTTP POST for /slave(1)/api/v1/executor from 172.31.23.107:49328 with User-Agent='scaleio/0.1'
      

      Then crashes out and the agent restarts with a new agent log:

      Log file created at: 2016/08/16 17:53:11
      Running on machine: ec2-52-38-65-6.us-west-2.compute.amazonaws.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      I0816 17:53:11.674993  4977 logging.cpp:194] INFO level logging started!
      I0816 17:53:11.678026  4977 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
      I0816 17:53:11.681545  4977 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
      I0816 17:53:11.682831  4977 main.cpp:434] Starting Mesos agent
      ...
      ...
      

        Attachments

        1. _usr_sbin_mesos-slave.0.crash
          1.81 MB
          David vonThenen

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                dvonthenen David vonThenen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: