Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-8612

Yarn application status is misreported for failed PySpark apps.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 1.3.0, 1.3.1, 1.4.0
    • Fix Version/s: None
    • Component/s: YARN
    • Labels:
      None
    • Environment:

      PySpark job run in yarn-client mode on CDH 5.4.2

      Description

      When a PySpark job fails, the YARN records and reports its status as successful. Hari Shreedharan pointed out to me that the ApplicationMaster records app success when system.exit is called. PySpark always exits by calling os._exit. Because of this, every PySpark application run on yarn is marked as completed successfully.

        Attachments

          Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-7736 Exception not failing Python applications (in yarn cluster mode)

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                juliet Juliet Hougland
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: