Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44591

Add jobTags to SparkListenerSQLExecutionStart

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0, 4.0.0
    • 3.5.0
    • Connect
    • None

    Description

      As part SPARK-43952, users can define job tags via SparkContext.addJobTag. These tags are then used to trigger cancelation via SparkContext.cancelJobsByTag. Furthermore, these tags can be used to logically group multiple jobs together.

      Listener of job events can retrieve job tags via SparkListenerJobStart.props.getProperty(SparkContext.SPARK_JOB_TAGS)

      Listener of SQL events can link SparkListenerJobStart & SparkListenerSQLExecutionStart via SparkListenerJobStart.props.getProperty(SQLExecution.EXECUTION_ID_KEY).

      However, some SQL executions do not trigger jobs (i.e. commands). As such listeners of SQL executions cannot resolve job tags of all executions.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jgauthier Jean-Francois Desjeans Gauthier
            jgauthier Jean-Francois Desjeans Gauthier
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment