Description
For jobs associated to SQL queries, it would be easier to understand the context to showing the SQL query in Job detail page.
Attachments
Issue Links
Activity
gatorsmile commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445899184
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445899666
**Test build #99926 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99926/testReport)** for PR 23068 at commit [`e7c2ebb`](https://github.com/apache/spark/commit/e7c2ebbda949918034cb9cb92ac6ef30af17d943).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
gatorsmile commented on a change in pull request #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#discussion_r240305749
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
##########
@@ -189,14 +189,19 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
require(parameterId != null && parameterId.nonEmpty, "Missing id parameter")
val jobId = parameterId.toInt
- val jobData = store.asOption(store.job(jobId)).getOrElse {
+ val (jobData, sqlExecutionId) = store.asOption(store.jobWithAssociatedSql(jobId)).getOrElseUnknown macro: { val content = <div id="no-info"> <p>No information to display for job {jobId}</p> </div> return UIUtils.headerSparkPage( request, s"Details for Job $jobId", content, parent) }+ val sqlDetailUrl = sqlExecutionId.map { id =>
Review comment:
Add a code comment to explain when `sqlExecutionId ` can be None.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
gatorsmile commented on a change in pull request #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#discussion_r240305948
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -56,6 +56,11 @@ private[spark] class AppStatusStore(
store.read(classOf[JobDataWrapper], jobId).info
}
+ def jobWithAssociatedSql(jobId: Int): (v1.JobData, Option[Long]) = {
Review comment:
Add a function description above this line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
gatorsmile commented on a change in pull request #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#discussion_r240306597
##########
File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
##########
@@ -56,6 +56,11 @@ private[spark] class AppStatusStore(
store.read(classOf[JobDataWrapper], jobId).info
}
+ def jobWithAssociatedSql(jobId: Int): (v1.JobData, Option[Long]) = {
Review comment:
Add a function description above this line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445901339
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445901351
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445901339
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445901351
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5933/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445983894
**Test build #99926 has finished(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99926/testReport)** for PR 23068 at commit [`e7c2ebb`](https://github.com/apache/spark/commit/e7c2ebbda949918034cb9cb92ac6ef30af17d943).
- This patch passes all tests.
- This patch merges cleanly.
- This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445899666
**Test build #99926 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99926/testReport)** for PR 23068 at commit [`e7c2ebb`](https://github.com/apache/spark/commit/e7c2ebbda949918034cb9cb92ac6ef30af17d943).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445984605
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445984613
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99926/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445984605
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-445984613
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99926/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094075
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094078
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5958/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094215
**Test build #99954 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99954/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094075
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094078
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5958/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446108007
**Test build #99954 has finished(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99954/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
- This patch *fails due to an unknown error code, -9*.
- This patch merges cleanly.
- This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446108049
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446108052
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99954/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446094215
**Test build #99954 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99954/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446108049
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446108052
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99954/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
gengliangwang commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446203181
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204346
**Test build #99977 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99977/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204383
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204391
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5976/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204383
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204391
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5976/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446284763
**Test build #99977 has finished(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99977/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
- This patch *fails Spark unit tests*.
- This patch merges cleanly.
- This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
SparkQA removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446204346
**Test build #99977 has started(https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99977/testReport)** for PR 23068 at commit [`0a63604`](https://github.com/apache/spark/commit/0a636049ecc721cdd31cd676fce79aeb6582dd7c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446285110
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins commented on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446285117
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99977/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446285110
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
AmplabJenkins removed a comment on issue #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#issuecomment-446285117
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99977/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
gatorsmile commented on a change in pull request #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068#discussion_r240750502
##########
File path: core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
##########
@@ -278,6 +279,17 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
<Strong>Status:</Strong>
</li>
+ {
+ if (sqlExecutionId.isDefined) {
+ <li>
+ <strong>Associated SQL Query: </strong>
+ {<a href={"%s/SQL/execution/?id=%s".format(
+ UIUtils.prependBaseUri(request, parent.basePath),
+ sqlExecutionId.get)
Review comment:
We might see a strange error if `sqlExecutionId ` is None.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
asfgit closed pull request #23068: SPARK-26098[WebUI] Show associated SQL query in Job page
URL: https://github.com/apache/spark/pull/23068
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
index bd3f58b6182c0..262ff6547faa5 100644
— a/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala
@@ -70,6 +70,8 @@ private[spark] class AppStatusListener(
private val liveTasks = new HashMap[Long, LiveTask]()
private val liveRDDs = new HashMap[Int, LiveRDD]()
private val pools = new HashMap[String, SchedulerPool]()
+
+ private val SQL_EXECUTION_ID_KEY = "spark.sql.execution.id"
// Keep the active executor count as a separate variable to avoid having to do synchronization
// around liveExecutors.
@volatile private var activeExecutorCount = 0
@@ -318,6 +320,8 @@ private[spark] class AppStatusListener(
val lastStageName = lastStageInfo.map(_.name).getOrElse("(Unknown Stage Name)")
val jobGroup = Option(event.properties)
.flatMap
+ val sqlExecutionId = Option(event.properties)
+ .flatMap(p => Option(p.getProperty(SQL_EXECUTION_ID_KEY)).map(_.toLong))
val job = new LiveJob(
event.jobId,
@@ -325,7 +329,8 @@ private[spark] class AppStatusListener(
if (event.time > 0) Some(new Date(event.time)) else None,
event.stageIds,
jobGroup,
- numTasks)
+ numTasks,
+ sqlExecutionId)
liveJobs.put(event.jobId, job)
liveUpdate(job, now)
diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
index b35781cb36e81..312bcccb1cca1 100644
— a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
@@ -56,6 +56,13 @@ private[spark] class AppStatusStore(
store.read(classOf[JobDataWrapper], jobId).info
}
+ // Returns job data and associated SQL execution ID of certain Job ID.
+ // If there is no related SQL execution, the SQL execution ID part will be None.
+ def jobWithAssociatedSql(jobId: Int): (v1.JobData, Option[Long]) =
+
def executorList(activeOnly: Boolean): Seq[v1.ExecutorSummary] = {
val base = store.view(classOf[ExecutorSummaryWrapper])
val filtered = if (activeOnly) {
diff --git a/core/src/main/scala/org/apache/spark/status/LiveEntity.scala b/core/src/main/scala/org/apache/spark/status/LiveEntity.scala
index 47e45a66ecccb..7f7b83a54d794 100644
— a/core/src/main/scala/org/apache/spark/status/LiveEntity.scala
+++ b/core/src/main/scala/org/apache/spark/status/LiveEntity.scala
@@ -64,7 +64,8 @@ private class LiveJob(
val submissionTime: Option[Date],
val stageIds: Seq[Int],
jobGroup: Option[String],
- numTasks: Int) extends LiveEntity {
+ numTasks: Int,
+ sqlExecutionId: Option[Long]) extends LiveEntity { var activeTasks = 0 var completedTasks = 0 @@ -108,7 +109,7 @@ private class LiveJob( skippedStages.size, failedStages, killedSummary) - new JobDataWrapper(info, skippedStages) + new JobDataWrapper(info, skippedStages, sqlExecutionId) }
}
diff --git a/core/src/main/scala/org/apache/spark/status/storeTypes.scala b/core/src/main/scala/org/apache/spark/status/storeTypes.scala
index ef19e86f3135f..eea47b3b17098 100644
— a/core/src/main/scala/org/apache/spark/status/storeTypes.scala
+++ b/core/src/main/scala/org/apache/spark/status/storeTypes.scala
@@ -68,7 +68,8 @@ private[spark] class ExecutorSummaryWrapper(val info: ExecutorSummary) {
*/
private[spark] class JobDataWrapper(
val info: JobData,
- val skippedStages: Set[Int]) {
+ val skippedStages: Set[Int],
+ val sqlExecutionId: Option[Long]) {
@JsonIgnore @KVIndex
private def id: Int = info.jobId
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala b/core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
index 55444a2c0c9ab..b58a6ca447edf 100644
— a/core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala
@@ -189,7 +189,7 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
require(parameterId != null && parameterId.nonEmpty, "Missing id parameter")
val jobId = parameterId.toInt
- val jobData = store.asOption(store.job(jobId)).getOrElse {
+ val (jobData, sqlExecutionId) = store.asOption(store.jobWithAssociatedSql(jobId)).getOrElseUnknown macro: { val content = <div id="no-info"> <p>No information to display for job {jobId}</p>@@ -197,6 +197,7 @@ private[ui] class JobPage(parent}+
{jobData.status}
val isComplete = jobData.status != JobExecutionStatus.RUNNING
val stages = jobData.stageIds.map { stageId =>
// This could be empty if the listener hasn't received information about the
@@ -278,6 +279,17 @@ private[ui] class JobPage(parent: JobsTab, store: AppStatusStore) extends WebUIP
<Strong>Status:</Strong></li>
+ {
+ if (sqlExecutionId.isDefined) {
+ <li>
+ <strong>Associated SQL Query: </strong>
+Unknown macro: {<a href={"%s/SQL/execution/?id=%s".format( + UIUtils.prependBaseUri(request, parent.basePath), + sqlExecutionId.get) + }>{sqlExecutionId.get}</a>}+ </li>
+ }
+ }
{
if (jobData.jobGroup.isDefined) {
<li>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
User 'gengliangwang' has created a pull request for this issue:
https://github.com/apache/spark/pull/23068