Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21541

Spark Logs show incorrect job status for a job that does not create SparkContext

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      If you run a spark job without creating the SparkSession or SparkContext, the spark job logs says it succeeded but yarn says it fails and retries 3 times. Also, since, Application Master unregisters with Resource Manager and exits successfully, it deletes the spark staging directory, so when yarn makes subsequent retries, it fails to find the staging directory and thus, the retries fail.

      Steps:
      1. For example, run a pyspark job without creating SparkSession or SparkContext.
      Example:
      import sys
      from random import random
      from operator import add
      from pyspark import SparkContext

      if _name_ == "_main_":
      print("hello world")

      2. Spark will mark it as FAILED. Got to the UI and check the container logs.

      3. You will see the following information in the logs:
      spark:
      7/07/14 13:22:10 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
      17/07/14 13:22:10 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED

      But yarn logs will show:
      2017-07-14 01:14:33,203 [AsyncDispatcher event handler] INFO attempt.RMAppAttemptImpl: appattempt_1493735952617_12443844_000001 State change from FINAL_SAVING to FAILED

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pgandhi Parth Gandhi
                Reporter:
                pgandhi Parth Gandhi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: