Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9868

NetworkInfo from the agent /state endpoint is not correct.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1
    • Fix Version/s: 1.6.3, 1.7.3, 1.8.2
    • Component/s: None
    • Sprint:
      Containerization: RI-16 Sp 50, Containerization: RI-16 51, Containerization: RI-17 52
    • Story Points:
      8

      Description

      NetworkInfo from the agent /state endpoint is not correct, which is also different from the networkInfo of /containers endpoint. Some frameworks rely on the state endpoint to get the ip address for other containers to run.

      agent's state endpoint

      {
      "state": "TASK_RUNNING",
      "timestamp": 1561574343.1521769,
      "container_status": {
      "container_id": {
      "value": "9a2633be-d2e5-4636-9ad4-7b2fc669da99",
      "parent": {
      "value": "45ebab16-9b4b-416e-a7f2-4833fd4ed8ff"
      }
      },
      "network_infos": [
      {
      "ip_addresses": [
      {
      "protocol": "IPv4",
      "ip_address": "172.31.10.35"
      }
      ]
      }
      ]
      },
      "healthy": true
      }
      

      agent's /containers endpoint

      "status": {
      "container_id": {
      "value": "45ebab16-9b4b-416e-a7f2-4833fd4ed8ff"
      },
      "executor_pid": 1723,
      "network_infos": [
      {
      "ip_addresses": [
      {
      "ip_address": "9.0.73.65",
      "protocol": "IPv4"
      }
      ],
      "name": "dcos"
      }
      ]
      }
      

      The ip addresses are different^^.

      The container is in RUNNING state and is running correctly. Just the state endpoint is not correct. One thing to notice is that the state endpoint used to show the correct IP. After there was an agent restart and master leader re-election, the IP address in the state endpoint was changed.

      Here is the checkpoint CNI network information

      OK-23:37:48-root@int-mountvolumeagent2-soak113s:/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S4/frameworks/26ffb84c-81ba-4b3b-989b-9c6560e51fa1-0171/executors/k8s-clusters.kc02__etcd__b50dc403-30d1-4b54-a367-332fb3621030/runs/latest/tasks/k8s-clusters.kc02__etcd-2-peer__5b6aa5fc-e113-4021-9db8-b63e0c8d1f6c # cat /var/run/mesos/isolators/network/cni/45ebab16-9b4b-416e-a7f2-4833fd4ed8ff/dcos/network.conf 
      {"args":{"org.apache.mesos":{"network_info":{"name":"dcos"}}},"chain":"M-DCOS","delegate":{"bridge":"m-dcos","hairpinMode":true,"ipMasq":false,"ipam":{"dataDir":"/var/run/dcos/cni/networks","routes":[{"dst":"0.0.0.0/0"}],"subnet":"9.0.73.0/25","type":"host-local"},"isGateway":true,"mtu":1420,"type":"bridge"},"excludeDevices":["m-dcos"],"name":"dcos","type":"mesos-cni-port-mapper"}
      
      OK-01:30:05-root@int-mountvolumeagent2-soak113s:/var/lib/mesos/slave/meta/slaves/60c42ab7-eb1a-4cec-b03d-ea06bff00c3f-S4/frameworks/26ffb84c-81ba-4b3b-989b-9c6560e51fa1-0171/executors/k8s-clusters.kc02__etcd__b50dc403-30d1-4b54-a367-332fb3621030/runs/latest/tasks/k8s-clusters.kc02__etcd-2-peer__5b6aa5fc-e113-4021-9db8-b63e0c8d1f6c # cat /var/run/mesos/isolators/network/cni/45ebab16-9b4b-416e-a7f2-4833fd4ed8ff/dcos/eth0/network.info
      {"dns":{},"ip4":{"gateway":"9.0.73.1","ip":"9.0.73.65/25","routes":[{"dst":"0.0.0.0/0","gw":"9.0.73.1"}]}}
      

        Attachments

          Activity

            People

            • Assignee:
              qianzhang Qian Zhang
              Reporter:
              gilbert Gilbert Song
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: