[SPARK-16595] Spark History server Rest Api gives Application not found error for yarn-cluster mode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.1
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Yesha Vora
• Created: 17/Jul/16 18:19
• Updated: 21/May/19 04:34
• Resolved: 21/May/19 04:34

Description

Scenario:

• Start SparkPi application in Spark1 using yarn-cluster mode (application_1468686376753_0041)
• After application finishes validate application exists in respective Spark History server.

Error loading url http://xx.xx.xx.xx:18080/api/v1/applications/application_1468686376753_0041/1/executors
HTTP Code: 404
HTTP Data: no such app: application_1468686376753_0041

spark HS log

16/07/16 15:55:29 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468678823755_0049.inprogress
16/07/16 15:56:20 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468678823755_0049
16/07/16 16:23:14 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468678823755_0061.inprogress
16/07/16 16:24:14 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468678823755_0061
16/07/16 17:42:32 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/local-1468690940553.inprogress
16/07/16 17:43:22 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/local-1468690940553
16/07/16 17:43:44 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/local-1468691017376.inprogress
16/07/16 17:44:34 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/local-1468691017376
16/07/16 18:53:10 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468686376753_0041_1.inprogress
16/07/16 19:03:26 INFO PackagesResourceConfig: Scanning for root resource and provider classes in the packages:
  org.apache.spark.status.api.v1
16/07/16 19:03:35 INFO ScanningResourceConfig: Root resource classes found:
  class org.apache.spark.status.api.v1.ApiRootResource
16/07/16 19:03:35 INFO ScanningResourceConfig: Provider classes found:
  class org.apache.spark.status.api.v1.JacksonMessageWriter
16/07/16 19:03:35 INFO WebApplicationImpl: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
16/07/16 19:03:36 INFO SecurityManager: Changing view acls to: spark
16/07/16 19:03:36 INFO SecurityManager: Changing modify acls to: spark
16/07/16 19:03:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/07/16 19:03:36 INFO ApplicationCache: Failed to load application attempt application_1468686376753_0041/Some(1)
16/07/16 19:04:21 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468686376753_0043.inprogress
16/07/16 19:12:02 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468686376753_0043
16/07/16 19:16:11 INFO SecurityManager: Changing view acls to: spark
16/07/16 19:16:11 INFO SecurityManager: Changing modify acls to: spark
16/07/16 19:16:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
16/07/16 19:16:11 INFO FsHistoryProvider: Replaying log path: hdfs://xx.xx.xx.xx:8020/spark-history/application_1468686376753_0043
16/07/16 19:16:22 INFO SecurityManager: Changing acls enabled to: false
16/07/16 19:16:22 INFO SecurityManager: Changing admin acls to:
16/07/16 19:16:22 INFO SecurityManager: Changing view acls to: hrt_qa

hdfs@xxx:/var/log/spark$ hdfs dfs -ls /spark-history/
Found 8 items
-rwxrwx---   3 hrt_qa hadoop      28793 2016-07-16 15:56 /spark-history/application_1468678823755_0049
-rwxrwx---   3 hrt_qa hadoop      28763 2016-07-16 16:24 /spark-history/application_1468678823755_0061
-rwxrwx---   3 hrt_qa hadoop   58868885 2016-07-16 18:59 /spark-history/application_1468686376753_0041_1
-rwxrwx---   3 hrt_qa hadoop   58841982 2016-07-16 19:11 /spark-history/application_1468686376753_0043
-rwxrwx---   3 hive   hadoop       5823 2016-07-16 11:38 /spark-history/local-1468666932940
-rwxrwx---   3 hive   hadoop       5757 2016-07-16 22:44 /spark-history/local-1468669677840.inprogress
-rwxrwx---   3 hrt_qa hadoop     484113 2016-07-16 17:43 /spark-history/local-1468690940553
-rwxrwx---   3 hrt_qa hadoop      57747 2016-07-16 17:44 /spark-history/local-1468691017376
hdfs@xxx:/var/log/spark$ hdfs dfs -ls /spark-history/application_1468686376753_0041_1
-rwxrwx---   3 hrt_qa hadoop   58868885 2016-07-16 18:59 /spark-history/application_1468686376753_0041_1

Issue Links

• duplicates: SPARK-15923 Spark Application rest api returns "no such app: <appId>"
• is duplicated by: SPARK-17111 Spark2 History server got "Failed to load application attempt"