Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9750

Agent V1 GET_STATE response may report a complete executor's tasks as non-terminal after a graceful agent shutdown

    XMLWordPrintableJSON

    Details

    • Sprint:
      Mesos Foundations: RI14 Sp 46, Mesos Foundations: RI14 Sp 47
    • Story Points:
      2

      Description

      When the following steps occur:
      1) A graceful shutdown is initiated on the agent (i.e. SIGUSR1 or /master/machine/down).
      2) The executor is sent a kill, and the agent counts down on executor_shutdown_grace_period.
      3) The executor exits, before all terminal status updates reach the agent. This is more likely if executor_shutdown_grace_period passes.

      This results in a completed executor, with non-terminal tasks (according to status updates).

      When the agent starts back up, the completed executor will be recovered and shows up correctly as a completed executor in /state. However, if you fetch the V1 GET_STATE result, there will be an entry in launched_tasks even though nothing is running.

      get_tasks {
        launched_tasks {
          name: "test-task"
          task_id {
            value: "dff5a155-47f1-4a71-9b92-30ca059ab456"
          }
          framework_id {
            value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
          }
          executor_id {
            value: "default"
          }
          agent_id {
            value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-S0"
          }
          state: TASK_RUNNING
          resources { ... }
          resources { ... }
          resources { ... }
          resources { ... }
          statuses {
            task_id {
              value: "dff5a155-47f1-4a71-9b92-30ca059ab456"
            }
            state: TASK_RUNNING
            agent_id {
              value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-S0"
            }
            timestamp: 1556674758.2175469
            executor_id {
              value: "default"
            }
            source: SOURCE_EXECUTOR
            uuid: "xPmn\234\236F&\235\\d\364\326\323\222\224"
            container_status { ... }
          }
        }
      }
      get_executors {
        completed_executors {
          executor_info {
            executor_id {
              value: "default"
            }
            command {
              value: ""
            }
            framework_id {
              value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
            }
          }
        }
      }
      get_frameworks {
        completed_frameworks {
          framework_info {
            user: "user"
            name: "default"
            id {
              value: "4b34a3aa-f651-44a9-9b72-58edeede94ef-0000"
            }
            checkpoint: true
            hostname: "localhost"
            principal: "test-principal"
            capabilities {
              type: MULTI_ROLE
            }
            capabilities {
              type: RESERVATION_REFINEMENT
            }
            roles: "*"
          }
        }
      }
      

      This happens because we combine executors and completed executors when constructing the response. The terminal task(s) with non-terminal updates appear under completed executors.
      https://github.com/apache/mesos/blob/89c3dd95a421e14044bc91ceb1998ff4ae3883b4/src/slave/http.cpp#L1734-L1756

        Attachments

          Activity

            People

            • Assignee:
              kaysoky Joseph Wu
              Reporter:
              kaysoky Joseph Wu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: