Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.2.0
-
None
Description
Hi, the Spark history server API "/application?status=running" returns applications whose latest attempts have completed (see example output below). Spark code says that an application is considered "running" when any one of the attempts are not "completed". https://github.com/apache/spark/blob/1cc34f3e58c92dd06545727e9d931008a1082bbf/core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala#L44
In my case, attempt 1 is shown as incomplete, but the attempt has finished and is stamped with an endTime. Is this a bug in Spark history server?
"attempts" : [ { "attemptId" : "2", "startTime" : "2018-02-14T23:59:17.785GMT", "endTime" : "2018-02-15T08:08:28.927GMT", "lastUpdated" : "2018-02-15T08:08:28.949GMT", "duration" : 29351142, "sparkUser" : {omitted}, "completed" : true, "startTimeEpoch" : 1518652757785, "endTimeEpoch" : 1518682108927, "lastUpdatedEpoch" : 1518682108949 }, { "attemptId" : "1", "startTime" : "2018-02-14T23:53:02.629GMT", "endTime" : "2018-02-14T23:59:13.426GMT", "lastUpdated" : "2018-02-23T05:03:45.434GMT", "duration" : 370797, "sparkUser" : {omitted}, "completed" : false, "startTimeEpoch" : 1518652382629, "endTimeEpoch" : 1518652753426, "lastUpdatedEpoch" : 1519362225434 } ]