Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-5536

Completed executors presented as alive

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.28.0
    • None
    • None
    • None
    • Ubuntu 14.04.3 LTS

    Description

      I'm running Mesos 0.28.0. Mesos slave(1)/state endpoint returns some completed executors not in frameworks.completed_executors but in frameworks.executors. Alsa this executor presents in monitor/statistics

      {
          "attributes": {...},
          "completed_frameworks": [],
          "flags": {...},
          "frameworks": [
              {
                  "checkpoint": true,
                  "completed_executors": [...],
                  "executors": [
                    {
                        "queued_tasks": [],
                        "tasks": [],
                        "completed_tasks": [
                            {
                                "discovery": {...},
                                "executor_id": "",
                                "framework_id": "f65b163c-0faf-441f-ac14-91739fa4394c-0000",
                                "id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
                                "labels": [...],
                                "name": "service",
                                "resources": {...},
                                "slave_id": "ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13",
                                "state": "TASK_KILLED",
                                "statuses": []
                            }
                        ],
                        "container": "ead42e63-ac92-4ad0-a99c-4af9c3fa5e31",
                        "directory": "...",
                        "id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
                        "name": "Command Executor (Task: service.a3b609b8-27ec-11e6-8044-02c89eb9127e) (Command: sh -c 'cd service...')",                  
                        "resources": {...},
                        "source": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e"                  
                    },
                    ...
                  ],
              }
          ],
          "git_sha": "961edbd82e691a619a4c171a7aadc9c32957fa73",
          "git_tag": "0.28.0",
          "version": "0.28.0",
          ...
      }
      
      "var/log/mesos/mesos-slave.INFO"
      13:33:19.479182  [slave.cpp:1361] Got assigned task service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      13:33:19.482566  [slave.cpp:1480] Launching task service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      13:33:19.483921  [paths.cpp:528] Trying to chown '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' to user 'mesosuser'
      13:33:19.504173  [slave.cpp:5367] Launching executor service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
      13:33:19.505537  [containerizer.cpp:666] Starting container 'ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' for executor 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework 'f65b163c-0faf-441f-ac14-91739fa4394c-0000'
      13:33:19.505734  [slave.cpp:1698] Queuing task 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' for executor 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      ...
      13:33:19.977483  [containerizer.cpp:1118] Checkpointing executor's forked pid 25576 to '/tmp/mesos/meta/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31/pids/forked.pid'
      13:33:35.775195  [slave.cpp:1891] Asked to kill task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      13:33:35.775645  [slave.cpp:3002] Handling status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000 f
      rom @0.0.0.0:0
      13:33:35.778105  [cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1) for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31
      13:33:35.778488  [disk.cpp:169] Updating the disk resources for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31 to cpus(*):0.1; mem(*):32
      13:33:35.780349  [mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 32MB for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e3
      1
      13:33:35.782573  [status_update_manager.cpp:320] Received status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a8
      1b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-9173
      9fa4394c-0000
      13:33:35.783860  [status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      13:33:35.788767  [slave.cpp:3400] Forwarding the update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000 to master@10.82.24.138:5050
      13:33:35.917932  [status_update_manager.cpp:392] Received status update acknowledgement (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      13:33:35.918143  [status_update_manager.cpp:824] Checkpointing ACK for status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000
      ...
      13:33:39.031054  [slave.cpp:2643] Got registration for executor 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@127.0.0.1:60083
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              janisz Tomasz Janiszewski
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: