Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30357

SparkContext: Invoking stop() from shutdown hook

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      pivotal kubernetes container service PKS  having 3master nodes and 3 worker nodes

       

      OS : ubuntu for k8-s cluster

       

       

      Description

      I'm getting below error while running spark-submit job in kubernetes , i didn't get any specific error

      19/12/26 07:38:18 INFO SparkContext: Invoking stop() from shutdown hook
      19/12/26 07:38:18 INFO SparkUI: Stopped Spark web UI at http://spark-ml-test-1577345808987-driver-svc.spark-jobs.svc:4040
      19/12/26 07:38:18 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
      19/12/26 07:38:18 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
      19/12/26 07:38:18 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
      19/12/26 07:38:18 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
      19/12/26 07:38:18 INFO MemoryStore: MemoryStore cleared
      19/12/26 07:38:18 INFO BlockManager: BlockManager stopped
      19/12/26 07:38:18 INFO BlockManagerMaster: BlockManagerMaster stopped
      19/12/26 07:38:18 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
      19/12/26 07:38:18 INFO SparkContext: Successfully stopped SparkContext
      19/12/26 07:38:18 INFO ShutdownHookManager: Shutdown hook called
      19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a/pyspark-665955c9-0dc6-49ee-9b57-0392be430f10
      19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /tmp/spark-fa047fa8-b573-44aa-ab65-e235fb0d5b09
      19/12/26 07:38:18 INFO ShutdownHookManager: Deleting directory /var/data/spark-ec98f5d3-022d-4eae-87aa-c23a96c77555/spark-b757b85f-3709-4215-8208-95db840a294a
      

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              v33ran00l Veera
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: