[HIVE-16984] HoS: avoid waiting for RemoteSparkJobStatus::getAppID() when remote driver died

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: Spark
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 28/Jun/17 19:07
• Updated: 29/Jun/17 23:16

Description

In HoS, after a RemoteDriver is launched, it may fail to initialize a Spark context and thus the ApplicationMaster will die eventually. In this case, there are two issues related to RemoteSparkJobStatus::getAppID():

1. Currently we call getAppID() before starting the monitoring job. For the first, it will wait for hive.spark.client.future.timeout, and for the latter, it will wait for hive.spark.job.monitor.timeout. The error message for the latter treats the hive.spark.job.monitor.timeout as the time waiting for the job submission. However, this is inaccurate as it doesn't include hive.spark.client.future.timeout.
2. In case the RemoteDriver suddenly died, currently we still may wait hopelessly for the timeouts. This should potentially be avoided if we know that the channel has closed between the client and remote driver.