Uploaded image for project: 'Stratos'
  1. Stratos
  2. STRATOS-1570

Fail to delete a pod in one cluster when it is a multi region deployment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.2
    • Fix Version/s: 4.1.3
    • Component/s: None
    • Labels:
      None

      Description

      Stratos fails to delete a pod from one of two clusters. This issue came with latest packs.
      The pod still exist in the cluster.
      TID: [0] [STRATOS] [2015-09-17 07:38:18,372] INFO

      {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} - Starting the moving of termination pending to obsolete for [member] tomcatr1.my-tomcat.tomcat.domain0fc0311f-36e9-4893-84b0-b16290326eae
      TID: [0] [STRATOS] [2015-09-17 07:38:18,372] INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
      • Found termination pending member and trying to move [member] tomcatr1.my-tomcat.tomcat.domain0fc0311f-36e9-4893-84b0-b16290326eae to obsolete list
        TID: [0] [STRATOS] [2015-09-17 07:38:18,372] INFO {org.apache.stratos.autoscaler.client.AutoscalerCloudControllerClient}
      • Terminating instance via cloud controller: [member] tomcatr1.my-tomcat.tomcat.domain0fc0311f-36e9-4893-84b0-b16290326eae
        TID: [0] [STRATOS] [2015-09-17 07:38:18,376] INFO {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Removing kubernetes pod: [application] tomcatr1 [cartridge] tomcat [member] tomcatr1.my-tomcat.tomcat.domain0fc0311f-36e9-4893-84b0-b16290326eae [pod] tomcatr1-pod-5
        TID: [0] [STRATOS] [2015-09-17 07:38:18,560] ERROR {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Could not delete kubernetes pod: [pod-id] tomcatr1-pod-5
        javax.ws.rs.WebApplicationException: HTTP 404 pods "tomcatr1-pod-5" not found
        at io.fabric8.kubernetes.api.ExceptionResponseMapper.fromResponse(ExceptionResponseMapper.java:40)
        at io.fabric8.kubernetes.api.ExceptionResponseMapper.fromResponse(ExceptionResponseMapper.java:31)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.checkResponse(ClientProxyImpl.java:302)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.handleResponse(ClientProxyImpl.java:725)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:683)
        at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:224)
        at com.sun.proxy.$Proxy83.deletePod(Unknown Source)
        at io.fabric8.kubernetes.api.KubernetesClient.deletePod(KubernetesClient.java:205)
        at org.apache.stratos.kubernetes.client.KubernetesApiClient.deletePod(KubernetesApiClient.java:145)
        at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.terminateContainer(KubernetesIaas.java:848)
        at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.terminateInstance(KubernetesIaas.java:136)
        at org.apache.stratos.cloud.controller.services.impl.InstanceTerminator.run(InstanceTerminator.java:56)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
        TID: [0] [STRATOS] [2015-09-17 07:38:18,561] WARN {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas}
      • Could not delete pod: [pod-id] tomcatr1-pod-5

      Problem occurs in the same cluster every time. However when I deploy only in the problematic cluster it works fine. Issue seems to occur when deployed in both clusters.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gayangk Gayan Gunarathne
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: