[SPARK-29637] SHS Endpoint /applications/<app_id>/jobs/ doesn't include description

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.4, 2.4.4, 3.0.0
• Fix Version/s: 2.4.5, 3.0.0
• Component/s: Spark Core
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 29/Oct/19 10:29
• Updated: 29/Oct/19 19:06
• Resolved: 29/Oct/19 19:06

Description

Starting from Spark 2.3, the SHS REST API endpoint /applications/<app_id>/jobs/  is not including description in the JobData returned. This is not the case until Spark 2.2.

Steps to reproduce:

• Open spark-shell

  scala> sc.setJobGroup("test", "job", false); 
  scala> val foo = sc.textFile("/user/foo.txt");
  foo: org.apache.spark.rdd.RDD[String] = /user/foo.txt MapPartitionsRDD[1] at textFile at <console>:24
  scala> foo.foreach(println);
  

• Access end REST API http://SHS-host:port/api/v1/applications/<app-id>/jobs/
• REST API of Spark 2.3 and above will not return description

Issue Links

• links to: GitHub Pull Request #26295