Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44591

Add jobTags to SparkListenerSQLExecutionStart

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0, 4.0.0
    • 3.5.0
    • Connect
    • None

    Description

      As part SPARK-43952, users can define job tags via SparkContext.addJobTag. These tags are then used to trigger cancelation via SparkContext.cancelJobsByTag. Furthermore, these tags can be used to logically group multiple jobs together.

      Listener of job events can retrieve job tags via SparkListenerJobStart.props.getProperty(SparkContext.SPARK_JOB_TAGS)

      Listener of SQL events can link SparkListenerJobStart & SparkListenerSQLExecutionStart via SparkListenerJobStart.props.getProperty(SQLExecution.EXECUTION_ID_KEY).

      However, some SQL executions do not trigger jobs (i.e. commands). As such listeners of SQL executions cannot resolve job tags of all executions.

      Attachments

        Activity

          People

            jgauthier Jean-Francois Desjeans Gauthier
            jgauthier Jean-Francois Desjeans Gauthier
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: