Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25222

Spark on Kubernetes Pod Watcher dumps raw container status

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0, 2.3.1
    • Fix Version/s: 3.0.0
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      Spark on Kubernetes provides logging of the pod/container status as a monitor of the job progress. However the logger just dumps the raw container status object leading to fairly unreadable output like so:

      18/08/24 09:03:27 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
      	 pod name: spark-groupby-1535101393784-driver
      	 namespace: default
      	 labels: spark-app-selector -> spark-47f7248122b9444b8d5fd3701028a1e8, spark-role -> driver
      	 pod uid: 88de6467-a77c-11e8-b9da-a4bf0128b75b
      	 creation time: 2018-08-24T09:03:14Z
      	 service account name: spark
      	 volumes: spark-local-dir-1, spark-conf-volume, spark-token-kjxkv
      	 node name: tab-cmp4
      	 start time: 2018-08-24T09:03:14Z
      	 container images: rvesse/spark:latest
      	 phase: Running
      	 status: [ContainerStatus(containerID=docker://23ae58571f59505e837dca40455d0347fb90e9b88e2a2b145a38e2919fceb447, image=rvesse/spark:latest, imageID=docker-pullable://rvesse/spark@sha256:92abf0b718743d0f5a26068fc94ec42233db0493c55a8570dc8c851c62a4bc0a, lastState=ContainerState(running=null, terminated=null, waiting=null, additionalProperties={}), name=spark-kubernetes-driver, ready=true, restartCount=0, state=ContainerState(running=ContainerStateRunning(startedAt=Time(time=2018-08-24T09:03:26Z, additionalProperties={}), additionalProperties={}), terminated=null, waiting=null, additionalProperties={}), additionalProperties={})]
      

      The LoggingPodStatusWatcher actually already includes code to nicely format this information but only invokes it at the end of the job:

      18/08/24 09:04:07 INFO LoggingPodStatusWatcherImpl: Container final statuses:
      
      
               Container name: spark-kubernetes-driver
               Container image: rvesse/spark:latest
               Container state: Terminated
               Exit code: 0
      

      It would be nice if we continually used the nice formatting throughout the logging.

      We already have patched this on our internal fork and will upstream a fix shortly.

        Attachments

          Activity

            People

            • Assignee:
              rvesse Rob Vesse
              Reporter:
              rvesse Rob Vesse
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: